UnicodeDecodeError: 'utf-8' codec can't decode byte 0xca in position 0: invalid continuation byte

tl;dr: You might need to change your Python Interpreter in the Pycharm settings. In my case (using pipenv), it was pointing to /Library/.../bin/Python with an uppercase P. Opening the folder and checking, I found that the file or symlink was actually python with a lowercase p. Changing it from .../Python to .../python in the Pycharm settings fixed this problem. (I had to restart Pycharm afterwards.) Screenshots below.

The longer version

While working on MembershipNerd today, I encountered a very annoying problem: I wasn’t able to run the server in Pycharm’s debug mode. This wasn’t happening outside of debugging, but when trying to run the debugger, it got the following errors:

/Users/redacted/.local/share/virtualenvs/membership-nerd-P7AnhCkV/bin/Python "/Users/redacted/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/212.5284.44/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py" --multiproc --qt-support=auto --client 127.0.0.1 --port 51729 --file /Users/redacted/development/membership-nerd/src/manage.py runserver 127.0.0.1:8000

Connected to pydev debugger (build 212.5284.44)
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/tokenize.py", line 330, in find_cookie
    line_string = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xca in position 0: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/tokenize.py", line 394, in open
    encoding, lines = detect_encoding(buffer.readline)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/tokenize.py", line 371, in detect_encoding
    encoding = find_cookie(first)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/tokenize.py", line 335, in find_cookie
    raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration for '/Users/redacted/.local/share/virtualenvs/membership-nerd-P7AnhCkV/bin/Python'
python-BaseException

Process finished with exit code 0

I searched Google for quite some time, but didn’t find any solution. Django didn’t seem to be the problem, because there were many people using flask with that same issue. The common thread between these problems seemed to be Pycharm/IntelliJ (obviously), and pipenv.

After comparing the command line that Pycharm runs when using a “normal” vs. a debug-configuration, I even had a look at their wrapper pydevd.py, because that was the major difference between those two modes. But that thing was just too complicated and since my debugger wasn’t working, I couldn’t simply step through it. 😑

The thing that was so strange with this error message was that it is complaining about an “invalid or missing encoding declaration” for the file .../bin/Python. But that file is a binary (or, when using pipenv on a Mac, a symlink to a binary). Why would the wrapper script try to determine the encoding for a binary file? That would only make for a python-script or some other text-file.

While I was wondering about that, I had a look at the folder where this file was supposed to be: /Users/redacted/.local/share/virtualenvs/membership-nerd-P7AnhCkV/bin/. Strangely enough, the folder did not contain a file named Python, only a file named python. Since the default file-system on a Mac is case-insensitive, that wasn’t a problem when using the normal run-configuration, but maybe Jetbrain’s pydevd.py-wrapper was stumbling over this?