Several options, none of them is trivial:
1. Remove the items as the user scrolls. Will not be simple to handle the fast flings.
2. Handle the ScrollChanged position and update the scroll position if the user goes back. Might work.
3. Use ViewsEx TouchPanelCreator to handle the touch event before it reaches the ScrollView. You can see a more complicated example in CLVNested. You will need to implement the scrolling movement yourself.