If you are implementing some fancy architecture, I advice you to implement a following test:
- Take batch of examples and run it through the network
- Take one (let’s say the third example form the previous batch) and run it through the network
- Compare the output of the example from batch to the output of a single example pass.
Ask anyone, these bugs happen. Happy debugging!