Python on Lin's Blog

How to Ignore Python Warnings

Sun, 12 Mar 2023 13:25:45 +0000

While writing Python, I sometimes run into warnings that either can't be solved or don't matter much. For example, while using OpenPyxl, it kept showing me: warn("Workbook contains no default style, apply openpyxl's default"). This warning tells me that OpenPyxl will automatically apply its default style to the file I load. And how do you fix it? You open the Excel file in Office and save it again. In my case this warning doesn't matter much, because my program uses only the contents of the Excel file, and the style has no effect on it.

When I see a warning, my habit is still to check and think about what's causing it. If I confirm it doesn't affect me, I'll ignore it in the code, rather than relying on the developer's memory that "this warning doesn't matter." It's not simply out of sight, out of mind. It's about building the good habit of "dealing with every warning I see."

How to Print a Progress Bar Effect in Python

Sat, 18 Feb 2023 16:16:32 +0000

A little trick I picked up a while ago while developing the checkout tool for our lab's snack corner. If you want something like a progress bar in the Terminal, that is, "print text in place and remove the old text," you can refer to this article (the same principle applies to other languages).

Why Python Prints Nothing When Piped to tee

Sat, 16 Jun 2018 01:12:08 +0000

Recently I wrote a Slack bot in Python. Being lazy, I just print the log messages directly and save them to a file with the tee command. That way I can see the bot’s activity in the console and keep the same information in a file. But when I combined Python’s print() with tee, I found that the log file often had no content, or messages that should have appeared showed up much later. After some research, I learned that Python’s print function checks where its output is going. If the output goes to a shell, it is written immediately. If it doesn’t, the output is stored in a buffer and only written out once the buffer is full. So when you use Python with tee, you run into this buffering problem, and the expected output doesn’t show up on the screen or in the file in time.

Generating a Datetime Series in Python

Sun, 30 Jul 2017 02:30:16 +0000

In Excel, you can generate a series of consecutive dates with a simple drag. In Python, you can do something similar with the datetime and timedelta modules. Whether you want intervals of one hour, one day, or one month, you can do it.

Python 3 CP950 Error When Printing to the Windows Command Prompt

Tue, 21 Feb 2017 23:59:09 +0000

Today, while teaching some juniors how to program, I tried to print a line of Chinese text to the Command Prompt with Python, but I got the error “UnicodeEncodeError: ‘cp950’ codec can’t encode character”. I tried all kinds of methods without success, and in the end I found the answer by searching online.

When experienced developers see this error, their first instinct is to assume the string contains a special character, so they immediately add the ignore parameter, but that does not help. When I checked, the character that caused the problem was a Japanese character. As everyone knows, Python 3 uses Unicode strings by default, so it should not have trouble printing anything. It’s not like we’re dealing with the notorious Big5 again…

Never Delete Items from a List Inside a Loop

Thu, 12 Jan 2017 20:35:34 +0000

“The foolish mistake I never get tired of stepping on, no matter how many times.” I always thought that once you make a mistake, you will remember it for good. But today I lost three hours to a bug at work. I could not figure out where the problem was. Finally, I carefully reread the code and realized… it was the same problem again. In one sentence: “Never delete any array element inside a loop.”

Python Global Exception Handler

Mon, 02 Jan 2017 15:00:12 +0000

Normally, exception handling should not be global, because when you apply exception handling globally, it becomes hard to pinpoint where the problem occurred and handle it. But in rare cases, you do need to write code like this. For example, this time I wrote a Python program to process webhooks. The information the server sends is very varied, so whenever an exception occurs, I needed to print out the environment data so I could debug it afterward. This is exactly when you should use the Global Exception Hook technique.

Python Join with Lambda - A More Customizable Join

Tue, 13 Dec 2016 16:05:52 +0000

If you write code, you’re surely familiar with split() and join(). In Python, you can use String.join(list) to merge a List into a string. For example, “,".join([“A”, “B”, “C”]) returns the string “A,B,C”. Going the other way, “A,B,C”.split(”,") splits it into a List. But have you ever run into needs like these that you couldn’t solve?

Joining strings while capitalizing the first letter of each one
The List holds numbers, but join needs strings to work with them

These problems can be solved with List Comprehension. For example, the type conversion part can be written like this:

nums = [1,2,3,4,5]
nums_str = [str(num) for num in nums]
print(",".join(nums_str))

But are there other solutions besides List Comprehension? I bet you've also thought about passing a lambda function to join, the way you can with sort!

Chinese Pinyin Conversion in Python (Translating Names and Addresses)

Fri, 09 Dec 2016 00:05:43 +0000

Today I helped a coworker with a large batch of name and address translation requests from Chinese to English, so I looked into how Python handles Hanyu Pinyin. After all, you can’t copy everything one by one into a post office form or a website, right? (lol). The result was surprisingly simple. Address translation can be handled through the Geocoder APIs of various providers, and for pinyin there are third-party libraries you can use. The translation quality is pretty good. It’s not perfect, but combined with some “manual intelligence” (工人智慧, a pun on 人工智慧, “artificial intelligence”), the small flaws are good enough. To fix the last small flaws, you would probably need to invest a lot of resources and technology and introduce machine learning methods. (I’m actually interested in trying that out someday, since there are really too few useful tools in the Chinese-speaking world.)

How to Make elasticsearch.py Fast (Bulk Operations)

Thu, 08 Dec 2016 00:57:01 +0000

When you work with Elasticsearch in Python, you will inevitably install the official elasticsearch.py library. Recently, while using Python to receive email logs and write them into Elasticsearch, I picked up quite a few lessons worth sharing. As everyone knows, AWS Lambda charges by execution time, so when writing Lambda functions every second counts. Anything that ran over 0.5 seconds had me in agony. So I went from inserting records one at a time with .index, to using streaming_bulk, and finally, unable to stand it anymore, mixing parallel_bulk with streaming_bulk. Here I will talk about the differences between these approaches and the pitfalls I ran into!

Converting a String to a Boolean in Python

Thu, 08 Dec 2016 00:18:21 +0000

Today, while writing an AWS Lambda function, I needed to read a setting from an environment variable and convert it from a string to a boolean so I could use it in a condition. The obvious guesses are that Python will convert it automatically, or that bool("True") will do the job. But first of all, Python does not do automatic type conversion. This is important! It simply raises an error without any fuss. But surely you’d expect bool() to solve it, right?! In fact, this landmine cost me half an hour today…

How to Use Python GroupBy on Data (List+Dict)

Wed, 07 Dec 2016 01:13:27 +0000

When working with data in Python, grouping by a certain key is a very common need. For example, when analyzing email logs, you might want to know how many emails were sent for each subject, or list the top five email providers for each subject. In a database, this is very simple. But what if you do not want to send the data to a database?

Besides Pandas, the powerful data processing library that can help with this, Python itself also has GroupBy built in!

What I Saw as an Attendee - PyCon Taiwan 2016

Thu, 23 Jun 2016 15:09:56 +0000

It has been almost a month since PyCon Taiwan 2016. I am very grateful to jserv for the ticket giveaway, which gave me the chance to attend this annual gathering of Taiwanese Python developers. After the event, I kept thinking about what angle to take for this write-up. In the past I mostly worked as a volunteer at community events. This time, since I was an attendee at PyCon, I decided to write a review from the attendee’s point of view, covering the good and the bad things I saw at this year’s PyCon Taiwan. The good parts are worth learning from for other conferences, and the bad parts are worth learning from too, so that they can be improved.

uWSGI

Fri, 24 Jul 2015 19:17:45 +0000

Introduction

uWSGI is a protocol, and it is also a program.

Why is it so complicated? Let’s start with the program. uWSGI was originally designed so that web applications written in Python could communicate with web servers such as Nginx and Apache. It uses its own uwsgi communication protocol. It also supports other protocols such as CGI, FastCGI, and sockets, so besides serving Python, it can also integrate with programming languages such as PHP and Go.

How to Manually Install Python on Linux Without Breaking yum

Fri, 19 Aug 2011 08:00:45 +0000

I previously wrote a program that automatically backs up important data on Linux and uploads it to Google Storage. It was written in Python 2.7, but the Python built into Linux is 2.4, and even if you upgrade with yum, the newest version it gives you is 2.4. So if you want a newer version of Python, you have to compile it yourself. Below, I’ll use Python 3.2 as the example.

Making Python Module Installation Easier: easy_install Tutorial and Egg File Installation Guide

Sun, 01 May 2011 16:50:04 +0000

As everyone knows, one of Python’s key features is its high extensibility. You can install third-party modules to get the functionality you want. For example, if you want to write code in Chinese, you can install the zhpy module. Installing these modules is not especially hard, but it is tedious. Some only need you to click Next a few times, while others require changing system settings. Here I want to introduce a program called “easy_install”, which makes installing modules for Python much simpler. It works similarly to yum on Linux, and with it you only need one command to install a module~ easy_install can also install packages in the so-called egg format.

[Python] The First Step into the Python World: HelloWorld (Printing Text and Writing Comments)

Sun, 20 Feb 2011 09:00:54 +0000

Almost every programming language book starts with a first practice program, which is Hello World. Hello World means making the program print "HelloWorld". In this article, we will show you how to write your first Python program and briefly explain some basic Python usage.

[Python] Python Interpreter Installation Tutorial

Fri, 18 Feb 2011 22:01:26 +0000

Recently I've been playing around with Google Storage, so I dusted off a Python book that had been gathering dust and read through it. Together with the documentation on Google Code, I wrote a small automatic backup program. It's still very simple, so I won't publish it yet. But writing it made me realize that Python is a fairly easy programming language. In terms of ease of learning and rigor, it's about on par with VB. In terms of verbosity, though, Python wins over VB (I mean VB is the wordier one XD). For example, an if statement in VB has to be written as if.....then.......end if, while in Python you just write if....: to get started. Indentation also matters a lot in Python. If the indentation is wrong, the program won't run, because the interpreter uses indentation to decide where selection structures, loops, and so on end. Personally, I'd recommend Python to beginners, since it lets you write very neat code ~