Hardly a software project today manages without Git. Once you have experienced the performance benefits, SVN will feel like something from the age of stagecoaches. Because GitHub has built a nice UI around this service, which is free and reliably stores your data, and because it's so easy to contribute through pull requests, many developers like myself swear by the San Francisco-based Git hoster.

Over the years, 57 publicly visible repositories have accumulated in my account; most contain CPAN modules, but more esoteric content like the text data for my blog usarundbrief.com is also stored there [1]. You can dig up some interesting facts by messing around in the associated metadata with Perl.

Especially in the context of automatic build systems, it's essential that access to the metadata in the Git repositories not be exclusively browser-based. Instead, automated scripts leverage APIs to retrieve everything you need from the cornucopia of data. To control this in Perl, the Net::GitHub collection of modules by Fayland Lam is available on CPAN. Equipped with the necessary access privileges, the API user can both access (Figure 1) and actively modify programs, say, by adding some new code.

Figure 1: The author page on GitHub lists your own repositories and contributions to other repositories. The API offers very similar abilities.

For GitHub to know who issues API requests and to intervene with corrective action where needed, an optional authentication token is included with each request to the REST API. Users can pick this up in the browser UI or from a script, as shown in Listing 1 [2]. It prompts you for your password, then uses SSL to send the data to the GitHub server, and receives an access token in exchange.

Instead of a password, the client then sends this token going forward and gets more generous access to the server. The access privileges that were set when the token was generated determine the token user's rights. For scripts to find the token, Listing 1 dumps it in YAML format into the ~/.githubrc file in the user's home directory. The command

$ ./token-get
Password: *****
/Users/mschilli/.githubrc written

carries out the necessary steps for the user stored in line 11. You'll have to adapt the string to your own GitHub account user ID.

You need to watch out for the note comment value; it must have different content for each newly requested token when you call the create_authorization() method. Otherwise, GitHub will refuse to issue the token and report an incorrect user/password combination as the cause. The real reason, however, is that GitHub lists the tokens keyed by this note value on the user's account page, where you can modify or revoke the tokens (Figure 2), and the notes must be unique for each token.

Figure 2: The GitHub page lists all previously generated access tokens.

Additionally, the scope parameter defines what rights the account owner grants to prospective clients. If it is left empty – as in Listing 1 – the server only allows read access to publicly available files [3]. In contrast, one or more entries like user or repo in the scope array later give users read/write access to data (e.g., to change their email address) or code commits.

Show Your ID

According to the terms of use [4], GitHub allows clients that authenticate up to 5,000 requests per hour, whereas anonymous requests are restricted to 60 per hour and IP. However, the search API that performs a pattern-based search in repository names or checked-in code is somewhat more generous; it allows 20 requests per minute with a token and five without. Figure 3 shows that the API also works without logging in – querying metadata from the GitHub mschilli/log4perl repository in this case.

Figure 3: Even without the auth token, the GitHub API reveals information about repos or users.

The number of requests permitted before the server slams the door is included in the HTTP header of the response. Figure 4 shows an example of the command without a token, in which seven queries were sent in the current hour, and 53 thus remain.

Figure 4: Sixty requests are allowed per hour; the client has another 53 left before the server resets the counter at the specified time.

The script in Listing 2 queries the account metadata of the GitHub user specified on the command line. The user authenticates with the access token, which the YAML module from CPAN extracted from the ~/.githubrc file. In addition to a huge mess of other fields, the number of publicly visible repositories created in the account is available in the returned JSON data; Listing 2 simply outputs their number in line 17:

$ ./repo-user mschilli
Mike Schilli owns 57 public repos
$ ./repo-user torvalds
Linus Torvalds owns 2 public repos

The repo-user script amazingly revealed that Linus Torvalds has created only two repositories on GitHub!

Torvalds 2 – Schilli 57

Lines 9 and 12 in Listing 2 grab the access token stored previously by Listing 1, and the method chain ->user->show() accesses the user's account information on GitHub while providing the token.

The example in Listing 3 determines which repositories an author already created on GitHub; this is done by extracting and outputting the repository name strings from the returned metadata.

Instead of using list(), you could use the list_user() method, which takes a username (Figure 5); if you pass it the string torvalds, you see that the Linux creator's two repositories are named linux and subsurface.

Figure 5: Listing 3 outputs all the user's GitHub repositories.

Winner on Points

If you want to know which repositories exist for a specific topic, you can find out by using the search API. For example, many years ago I launched a project named Log4perl and published it on GitHub. Listing 4 now searches for all repositories on GitHub, whose names contain the log4perl pattern. Surprisingly, the script returns a whopping 117 results:

$ ./repo-list-multi
mschilli/log4perl (84.5)
cowholio4/log4perl_gelf (15.2)
TomHamilton/Log4Perl (14.1)
lammel/moosex-log-log4perl (9.7)
...
$ eg/repo-list-multi | wc -l
117

GitHub limits the number of results returned by the server to 100 by default. If you want more, you can increase the pagination size of the returned data with the per_page parameter. Or, as shown in Listing 4, you can ask whether there are more results for the query after the first packet of 100 has been returned, using has_next_page(). If this is the case, a subsequent call to next_page() dumps the next cartful of data into the search object (see Listing 4).

The script sets the sort search parameter in line 14 to stars, together with a value of desc (for descending) so that the result is sorted by popularity of matching repositories. If you are interested in a project on GitHub, you can click on its star icon to keep up with its progress. The number of stars on a given project is an indication of how popular a project is. In a query result, the number of stars on the project is given in the stargazers_count field, and the name of the project can be found in full_name.

If you happen to use a local GitHub Enterprise installation instead of the public github.com, you can insert a leading third parameter pair,

api_url => https://.../api/v3

to the new() method of the Net::GitHub class; then, you will receive information from this source instead.

If you are looking for patterns in the checked-in code, Listing 5 gives you a taste of what the GitHub API offers in this area. The search() method elicits a search object from the GitHub object; the search object's code() method will in turn initiate a search in the repository code. The query

snickers in:file language:perl repo:mschilli/log4perl

searches files that contain Perl code in the mschilli/log4perl repository for the word snickers. The results,

$ ./repo-search
lib/Log/Log4perl.pm
lib/Log/Log4perl/Config.pm

show that the code in the Log4perl repository has two files with occurrences of the word snickers and prove that the author not only uses the names of tasty candy bars in programming examples but also has a tendency toward oddball humor.