Python

While writing Python, I sometimes run into warnings that either can't be solved or don't matter much. For example, while using OpenPyxl, it kept showing me: warn("Workbook contains no default style, apply openpyxl's default"). This warning tells me that OpenPyxl will automatically apply its default style to the file I load. And how do you fix it? You open the Excel file in Office and save it again. In my case this warning doesn't matter much, because my program uses only the contents of the Excel file, and the style has no effect on it. When I see a warning, my habit is still to check and think about what's causing it. If I confirm it doesn't affect me, I'll ignore it in the code, rather than relying on the developer's memory that "this warning doesn't matter." It's not simply out of sight, out of mind. It's about building the good habit of "dealing with every warning I see." ...

A little trick I picked up a while ago while developing the checkout tool for our lab's snack corner. If you want something like a progress bar in the Terminal, that is, "print text in place and remove the old text," you can refer to this article (the same principle applies to other languages). ...

Recently I wrote a Slack bot in Python. Being lazy, I just print the log messages directly and save them to a file with the tee command. That way I can see the bot’s activity in the console and keep the same information in a file. But when I combined Python’s print() with tee, I found that the log file often had no content, or messages that should have appeared showed up much later. After some research, I learned that Python’s print function checks where its output is going. If the output goes to a shell, it is written immediately. If it doesn’t, the output is stored in a buffer and only written out once the buffer is full. So when you use Python with tee, you run into this buffering problem, and the expected output doesn’t show up on the screen or in the file in time. ...

In Excel, you can generate a series of consecutive dates with a simple drag. In Python, you can do something similar with the datetime and timedelta modules. Whether you want intervals of one hour, one day, or one month, you can do it. ...

Today, while teaching some juniors how to program, I tried to print a line of Chinese text to the Command Prompt with Python, but I got the error “UnicodeEncodeError: ‘cp950’ codec can’t encode character”. I tried all kinds of methods without success, and in the end I found the answer by searching online. When experienced developers see this error, their first instinct is to assume the string contains a special character, so they immediately add the ignore parameter, but that does not help. When I checked, the character that caused the problem was a Japanese character. As everyone knows, Python 3 uses Unicode strings by default, so it should not have trouble printing anything. It’s not like we’re dealing with the notorious Big5 again… ...

“The foolish mistake I never get tired of stepping on, no matter how many times.” I always thought that once you make a mistake, you will remember it for good. But today I lost three hours to a bug at work. I could not figure out where the problem was. Finally, I carefully reread the code and realized… it was the same problem again. In one sentence: “Never delete any array element inside a loop.” ...

Normally, exception handling should not be global, because when you apply exception handling globally, it becomes hard to pinpoint where the problem occurred and handle it. But in rare cases, you do need to write code like this. For example, this time I wrote a Python program to process webhooks. The information the server sends is very varied, so whenever an exception occurs, I needed to print out the environment data so I could debug it afterward. This is exactly when you should use the Global Exception Hook technique. ...

If you write code, you’re surely familiar with split() and join(). In Python, you can use String.join(list) to merge a List into a string. For example, “,".join([“A”, “B”, “C”]) returns the string “A,B,C”. Going the other way, “A,B,C”.split(”,") splits it into a List. But have you ever run into needs like these that you couldn’t solve? Joining strings while capitalizing the first letter of each one The List holds numbers, but join needs strings to work with them These problems can be solved with List Comprehension. For example, the type conversion part can be written like this: nums = [1,2,3,4,5] nums_str = [str(num) for num in nums] print(",".join(nums_str)) But are there other solutions besides List Comprehension? I bet you've also thought about passing a lambda function to join, the way you can with sort!...

Today I helped a coworker with a large batch of name and address translation requests from Chinese to English, so I looked into how Python handles Hanyu Pinyin. After all, you can’t copy everything one by one into a post office form or a website, right? (lol). The result was surprisingly simple. Address translation can be handled through the Geocoder APIs of various providers, and for pinyin there are third-party libraries you can use. The translation quality is pretty good. It’s not perfect, but combined with some “manual intelligence” (工人智慧, a pun on 人工智慧, “artificial intelligence”), the small flaws are good enough. To fix the last small flaws, you would probably need to invest a lot of resources and technology and introduce machine learning methods. (I’m actually interested in trying that out someday, since there are really too few useful tools in the Chinese-speaking world.) ...

When you work with Elasticsearch in Python, you will inevitably install the official elasticsearch.py library. Recently, while using Python to receive email logs and write them into Elasticsearch, I picked up quite a few lessons worth sharing. As everyone knows, AWS Lambda charges by execution time, so when writing Lambda functions every second counts. Anything that ran over 0.5 seconds had me in agony. So I went from inserting records one at a time with .index, to using streaming_bulk, and finally, unable to stand it anymore, mixing parallel_bulk with streaming_bulk. Here I will talk about the differences between these approaches and the pitfalls I ran into! ...