Jupyter: RuntimeError: This event loop is already running
I’ve been using the twint library to explore the Neo4j twitter community, and ran into an initially confusing error when I moved the code I’d written into a Jupyter notebook.
The first three cells of my notebook contain the following code:
Cell 1:
! pip install twint
Cell 2:
import json
import twint
Cell 3:
users = ["vikatakavi11", "tee_mars3"]
for username in users[:10]:
c = twint.Config()
c.Username = username
c.User_full = False
c.Store_object = True
c.Hide_output = True
twint.run.Followers(c)
followers = twint.output.follow_object
if not username in followers:
followers[username] = {"followers": []}
print(followers)
twint.output.follow_object = {}
If we execute those three cells we’ll get the following error:
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-5-796d99d0022c> in <module>
6 c.Hide_output = True
7
----> 8 twint.run.Followers(c)
9 followers = twint.output.follow_object
10
~/projects/twitter-graph/a/src/twint/twint/run.py in Followers(config)
219 config.Favorites = False
220 config.TwitterSearch = False
--> 221 run(config)
222 if config.Pandas_au:
223 storage.panda._autoget("followers")
~/projects/twitter-graph/a/src/twint/twint/run.py in run(config, callback)
198 def run(config, callback=None):
199 logme.debug(__name__+':run')
--> 200 get_event_loop().run_until_complete(Twint(config).main(callback))
201
202 def Favorites(config):
/usr/lib/python3.6/asyncio/base_events.py in run_until_complete(self, future)
458 future.add_done_callback(_run_until_complete_cb)
459 try:
--> 460 self.run_forever()
461 except:
462 if new_task and future.done() and not future.cancelled():
/usr/lib/python3.6/asyncio/base_events.py in run_forever(self)
412 self._check_closed()
413 if self.is_running():
--> 414 raise RuntimeError('This event loop is already running')
415 if events._get_running_loop() is not None:
416 raise RuntimeError(
RuntimeError: This event loop is already running
I knew that the twint library was using an event loop, but I didn’t realise that this would cause a problem with Jupyter until I read this GitHub issue, which explains that the Jupyter kernel is also running an event loop, and these loops can’t be nested.
This thread (https://github.com/jupyter/notebook/issues/3397#issuecomment-376803076) explains that we have two event loops running - one in Jupyter and one in the twint library. Luckily Ewald de Wit created an awesome library called 'nest_asyncio' that we can use to workaround this error.
I added the following cells:
Cell 4:
!pip install nest_asyncio
And now we can apply this library:
import nest_asyncio
nest_asyncio.apply()
And finally, we’ll try to find the followers for our users again:
for username in users[:10]:
c = twint.Config()
c.Username = username
c.User_full = False
c.Store_object = True
c.Hide_output = True
twint.run.Followers(c)
followers = twint.output.follow_object
if not username in followers:
followers[username] = {"followers": []}
print(followers)
twint.output.follow_object = {}
And this time it works!
{'vikatakavi11': {'followers': ['ganimohamed7861', 'ganimohamed7861']}}
{'tee_mars3': {'followers': ['christophe_cruz', 'leclercq_ub']}}
The full code for the notebook is in the gist below:
About the author
I'm currently working on short form content at ClickHouse. I publish short 5 minute videos showing how to solve data problems on YouTube @LearnDataWithMark. I previously worked on graph analytics at Neo4j, where I also co-authored the O'Reilly Graph Algorithms Book with Amy Hodler.