My favorite NN debugging test

If you are implementing some fancy architecture, I advice you to implement a following test:

1. Take batch of examples and run it through the network
2. Take one (let’s say the third example form the previous batch) and run it through the network
3. Compare the output of the example from batch to the output of a single example pass. 

Ask anyone, these bugs happen. Happy debugging!