This blog post write-up is both for my own future reference and for anyone else who might find it helpful.

Contents

• Introduction
• Adding functionality with custom modules
• The View
  • Outputting the HTML via the Fields section
  • Best practice
  • Filtering the View output
  • Accessing feed custom fields with Relationships
  • Add some AJAX
• Refreshing the feeds
• Conclusion

Introduction

The core functionality is based on a View and the Aggregator module. The site also uses the Rate and Voting API modules to enable users to ‘react’ to any news item by tapping on one of the emojis. The theme is a subtheme of Olivero, which ships with Drupal core. I quickly set up Drupal using SiteGround’s app installer. (Handily, the install comes with Composer, Drush and git out of the box.)

Adding functionality with custom modules

As well as contributed modules Aggregator and Rate, there are three custom modules:

‘Cookie Voter’

Cookie Voter alters the Rate module so that it uses cookies instead of IP addresses to track anonymous emoji reactions. Using IP addresses can cause users on shared networks to see others’ reactions appear as their own, which is confusing. BuzzFeed takes a similar approach, storing user reactions in the browser’s localStorage. On the other hand, the Bear blogging platform appears to use IP addresses to record upvotes. In Bear’s case that’s a sensible approach as votes affect a list of trending posts. At AnimalRights.fyi, emoji reactions simply offer users an informal way to engage with news stories.

‘Custom Headline Filter’

Custom Headline Filter provides a Views filter for removing duplicate (or overly similar) headlines. Such headlines can appear due to malformation of the incoming RSS feed, crossposting between websites, or when multiple news outlets report on the same story. There is a similarity threshold which can be adjusted to taste in the Views UI. The similarity is calculated by PHP’s similar_text function.

‘Custom Twig Extensions’

I needed to use PHP’s preg_replace function, and wanted to avoid installing the Twig Extensions module; so I created Custom Twig Extensions. (I try to install as few modules as possible to keep the complexity at a minimum and make site maintenance easier.)

Custom modules summary

All three modules were basically written by Claude (3.5 Sonnet), though there was a fair amount of back and forth between it and myself. In the case of Cookie Voter, for example, I asked it to refactor this Drupal 7 code for Drupal 8+, but it took much prompting to achieve the desired result. ‘We’ eventually landed on a solution after I fed the AI some relevant code from the Rate module codebase. This is a useful tip I will bear in mind when problem solving with a chatbot in future: don’t assume it already ‘knows’ a given codebase. (I had a conversation with ChatGPT about why supplying codebase excerpts for more obscure coding problems might be necessary.)

The View

Here’s a screenshot of the View, which is where the user-facing page is built:

Outputting the HTML via the Fields section

The Fields section forms the HTML output of the View, with Twig as the templating language. Here’s the Fields section aggregated into a single template representation:

<div class="news-item {{ title_1 }} fade-in-quick">
  <h3 class="headline{% if field_podcast == '1' %} icon-podcast{% endif %}" iid="{{ iid }}"><a href="{{ link }}" target="_blank">{{ title }}</a></h3>

  {# Get first paragraph of description and remove HTML tags #}
  {% set first_paragraph = description|split('</p>')[0]|trim %}
  {% set first_paragraph = first_paragraph starts with '<p>' ? first_paragraph[3:] : first_paragraph %}
  {% set first_paragraph = first_paragraph|striptags|trim %}

  {# Handle truncation #}
  {% set last_char = first_paragraph|last %}
  {% set last_three = first_paragraph|slice(-3) %}
  {% set last_nine = first_paragraph|slice(-9) %}

  {%
    if last_char not in ['.', '!', '?', '…', ':'] and
    last_three != '[…]' and
    first_paragraph|slice(-2) != '."' and
    first_paragraph|slice(-6) != '&nbsp;' and
    last_nine != ' ... more'
  %}
    {% set first_paragraph = first_paragraph ~ '.' %}
  {% elseif last_char == '…' %}
    {% set first_paragraph = first_paragraph|custom_replace('/(?<!\s)…$/u', ' […]') %}
  {% elseif last_nine == ' ... more' %}
    {% set first_paragraph = first_paragraph|custom_replace('/ \.\.\. more$/', ' […]') %}
  {% elseif last_three == '...' %}
    {% set first_paragraph = first_paragraph|custom_replace('/(?<!\s)\.{3}$/u', ' […]') %}
  {% elseif last_char == ':' %}
    {% set first_paragraph = first_paragraph|custom_replace('/:$/', '.') %}
  {% endif %}

  {# Hide descriptions that include these strings #}
  {% set excluded_patterns = [
    '©',
    'Image courtesy of',
    'Image supplied by',
    'Image credit',
    'The post',
    'Image:',
    'No abstract',
    'If you enjoyed this episode',
    'Published on'
  ] %}
  {% set is_excluded = excluded_patterns|filter(pattern => pattern in first_paragraph)|length > 0 %}

  {% if first_paragraph|length > 5 and not is_excluded %}
  <div class="views-field views-field-description">
    {{ first_paragraph|custom_replace('/&nbsp;/', '')|custom_replace('/&amp;/', '&') }}
    {% if field_include_feed_description == '1' %}
      {{ description_1 }}
    {% endif %}
  </div>
  {% endif %}

  <div class="meta">via {{ field_website }}
  {% if field_donate %}
    <span>
      <span class="time">{{ timestamp }}</span><span> {{ field_donate }}</span>
    </span>
  {% else %}
    <span class="time">{{ timestamp }}</span>
  {% endif %}
  </div>
</div>

<div class="comment fade-in-quick">{{ field_comment }}</div>

Most of the logic here is a bunch of heuristics which tidy up the HTML contained within the RSS feeds, eg standardising the ellipsis style for truncated descriptions, and removing descriptions that don’t provide any value to the reader.

You’ll see in the screenshot that a number of fields are set to ‘hidden’. Hiding a field makes its value available to output in the ‘Rewrite results’ section of subsequent fields. This allows you to combine or perform logic on two or more fields at once. The Field ‘Aggregator feed item: Title’, for instance, uses both the ‘Link’ and ‘Podcast?’ fields. The following markup is from the ‘Title’ field configuration under ‘Rewrite results’:

<div class="news-item {{ title_1 }} fade-in-quick">
  <h3 class="headline{% if field_podcast == '1' %} icon-podcast{% endif%}" iid="{{ iid }}"><a href="{{ link }}" target="_blank">{{ title }}</a></h3>

Here’s a screenshot of the UI for the ‘Title’ field:

Best practice

It’s probably better practice to create a Twig template file override instead of scattering the template across the View’s GUI as I’ve done here. Using the GUI is great for quickly prototyping Views, but you may later want to port it to a Twig template file so you can see the full template at a glance. (I may do this for my View as part of a project refinement exercise.)

Having all the code in a single template file also makes it easy to track changes. That said, it’s still possible to track changes with the GUI approach by exporting the configuration (drush cex) after making changes then committing the output to git.

Filtering the View output

The ‘Filter criteria’ section removes certain items entirely, such as sponsored posts and recipes. ‘Filter headlines (exposed)’ provides a text field by the which the user can search to filter news items by their headline.

Accessing feed custom fields with Relationships

The View is set up to list Aggregator feed items specifically; not the actual feeds. The ‘Aggregator feed’ relationship (under Advanced in the top-right of the Views screenshot) allows us to include custom fields from Aggregator feeds themselves. These include ‘Link’ fields for the feed’s website and a page where the user can donate to or otherwise support the website. Here’s a screenshot of /admin/config/services/aggregator/fields:

The ‘field_aggregator_item_rss_item_metadata’ Relationship links the View to a custom content type called ‘RSS Item Metadata’, which allows us to attach additional metadata to individual feed items. An ‘Entity reference’ field allows us to search for the news item we want. We can also add a comment beneath a news item or pin it to the right-hand column. Here’s the ‘Manage fields’ UI:

Add some AJAX

Make sure ‘Use AJAX’ (under ‘Other’) is set to ‘Yes’. This allows for pagination and filtering (‘Filter by headline’) without reloading the entire page.

Refreshing the feeds

I’ve set the cron to run every 10 minutues, which will update the feeds with any new items. The cron interval is set via the directive $config['automated_cron.settings']['interval'] = 600; in settings.php. (At /admin/config/system/cron, the equivalent field in the UI – ‘Run cron every’ – is set to ‘Never’, but Drupal will ignore this value.)

There is an ‘Update interval’ field on each feed (/aggregator/sources/FEED_ID/configure), which I set to ‘15 mins’, ‘1 hour’, ‘1 day’ etc., depending on how frequently the given website tends to post new content. When the cron runs, it checks when each feed was last refreshed, and if the time elapsed since the last refresh exceeds the update interval, the feed will be fetched again, and any new items will be displayed by the View.

Additionally, news items older than a year are deleted, via another directive in settings.php: $config['aggregator.settings']['items']['expire'] = 31536000;. (The ‘Discard items older than’ field at /admin/config/services/aggregator/settings is set to ‘Never’, and, like ‘Run cron every’, is overridden by settings.php.)

Conclusion

This post is an overview rather than a step-by-step tutorial. I may explore specific aspects of the website’s functionality in more depth in future posts.

Concerning Drupal as a platform, I wouldn’t necessarily recommend it for side projects like this. Drupal is complex, which is fine because it’s powerful; but managing the system (mainly running regular updates and managing configuration) can be a massive ball-ache. I only chose it for AnimalRights.fyi because I already build Drupal sites professionally. (And LLMs like Claude and ChatGPT make working with Drupal and similarly complex platforms much easier.)

Now that I have ZX Spectrum hardware for the first time in a couple of decades, I thought it would be a good time to see whether I could create a ‘Hello, world’ program in Z80 assembly language, convert it to machine code, and run it.

Reader caution: I don’t really know what I’m doing here. This blog post is more or less personal documentation. I’ve tried to avoid inaccuracies, but feel free to correct me in the comments.

I found the take on ‘Hello, world’ described on Benjamin Blundell’s website to be nice and succinct, so I used that. Here it is:

org $8000
  ld bc, MY_STRING

MY_LOOP:
  ld a, (bc)
  cp 0
  jr z, END_PROGRAM
  rst $10
  inc bc
  jr MY_LOOP

END_PROGRAM:
  ret

MY_STRING:
  defb "Hello, world!"
  defb 13, 0

I’ll use the Pasmo assembler to translate the instructions into binary machine code so that the Spectrum can understand it.

Some notes, based on Benjamin’s blog post, on what each line does:

org $8000

This is an assembler directive to set the starting (‘origin’) memory location (address) of the proceeding program ($8000 is 0x8000, or 8000 in hex, or decimal 32768). This location is safely within a 48K Spectrum’s non-system memory. (See Dean Belfield’s website for details on how memory is mapped in the both 48 and 128K models.)

ld bc, MY_STRING

Load register pair BC with the starting address of MY_STRING. Registers are small storage areas on the processor where data can be manipulated. We need to move a given value stored in memory into a register before anything can be done with it (eg print it on screen, do a calculation with it). We don’t need to think about this when working in a higher-level language like BASIC or JavaScript: memory management is handled for us. However when writing assembly language we need to move values explicitly between memory and the registers.

Regarding the address of our string, you may be wondering, as I did: ‘Hello, world!’ isn’t actually written to memory until later in the code, so how does the assembler know at this stage what the address is? The answer is that most assemblers, including Pasmo, run two passes, the first of which will read through the assembly code to determine the address of each label. Only on the second pass will the machine code be generated. So when the instruction ld bc, MY_STRING is encountered on pass two, the assembler knows the starting address of MY_STRING. This concept and process is referred to as ‘forward referencing’.

MY_LOOP:

Add label MY_LOOP to mark a loop that will cycle through each byte of whatever is stored starting at location MY_STRING.

ld a, (bc)

Load register A (the accumulator) with the value of the first byte at address MY_STRING (which will be the ‘H’ from ‘Hello, world!’ on the first go-around of this loop).

cp 0

Compare the contents of register A with 0. 0 is the last byte, after Hello, world! and a carriage return, set by the instruction defb 13, 0 later.

jr z, END_PROGRAM

Jump to END_PROGRAM if the above comparison is true (ie the value in A is 0).

rst $10

Call the ROM routine at address 0x10, which prints whatever is in register A to the screen. RST (or ‘restart’) is the same as CALL, except it uses only one byte, and is faster. However, you can only use it with eight specific ROM addresses, one of which is 0x10, which we call here. (Source: ‘Spectrum Machine Language for the Absolute Beginner’, page 128.) You could think of these routines as built-in helper functions which perform a specific task.

inc bc

Increment register pair BC so it moves to the next byte in memory (ie the next character in our string).

jr MY_LOOP

Jump back to the top of the loop.

END_PROGRAM:

Another label. Labels can have arbitrary values, and help us and the assembler navigate the code.

ret

Exit program.

defb "Hello, world!"

Define bytes with the string we want to print.

defb 13, 0

Define two more bytes — a carriage return (13) and string terminator (0) — so that cp 0 earlier in the code can check whether we’ve reached the end of the string.

I saved the file as helloworld.asm.

Converting our program into machine code

We’ll use the command line tool Pasmo (a Z80 cross-assembler) to assemble our machine code and create our .tap file, which we can then run on a Spectrum emulator or actual Speccy hardware. The TAP will comprise helloworld.asm in machine code form, and a BASIC loader program. The latter loads the machine code into memory and runs it. (Alternative assemblers include zasm, and Odin and Zeus if you’d like to develop on an actual Spectrum.)

Build and install Pasmo

I’m using macOS here, but the steps should be similar on other platforms.

1. First install CMake, a C++ build tool. I’ll use Homebrew: brew install cmake
2. Download Pasmo: git clone https://github.com/jounikor/pasmo.git
3. cd pasmo/pasmo (steps 3 to 7 are taken from the Pasma README.md)
4. mkdir build
5. cd build
6. cmake ../
7. make
8. Copy Pasmo to /usr/local/bin/ so that you can run it from anywhere: sudo cp pasmo /usr/local/bin/
9. Relaunch the shell: exec zsh -l
10. Verify that you can run Pasmo by viewing the manual page: pasmo man

Assemble our program and create the .tap file

1. Before we run the assembly process, add the Pasmo directive END $8000 at the end of your helloworld.asm file. This will prompt Pasmo to include a RANDOMIZE USR statement in the BASIC loader program. This will ensure our machine code runs when we launch the .tap file
2. Run pasmo --tapbas helloworld.asm helloworld.tap

Open helloworld.tap in Fuse or another emulator, or on an actual Spectrum Next, as I did. It should look like this:

The BASIC loader program

This is the BASIC loader program that Pasmo created:

10 CLEAR 32767
20 POKE 23610,255
30 LOAD "" CODE
40 RANDOMIZE USR 32768

Line by line:

• 10 CLEAR 32767 — ensure that the BASIC interpreter doesn’t write to memory above address 32767 (as well as clear any variables that are already stored there). This value should be a byte before the start of where our machine code will be stored.
• 20 POKE 23610,255 — ‘avoid a[n] error message when using +3 loader’. (Source: spectrum.cxx in the pasmo-0.5.5 codebase.)
• 30 LOAD "" CODE — load the next binary code file the Spectrum finds (which would typically have been on a cassette tape back in the day, located just after the BASIC loader program).
• 40 RANDOMIZE USR 32768: call (run) the machine code that’s stored starting at address 32768, which we specified in our helloworld.asm code on line 1 (org $8000), and at the end (END $8000) as Pasmo requires. $8000 is a shorthand for hexadecimal 8000 (0x8000), or decimal 32768.

What next?

I just received my copy of 40 Best Machine Code Routines for the ZX Spectrum by John Hardman and Andrew Hewson (with a new chapter on the Next by Jim Bagley), and it seems like an accessible guide to using machine code. I’ll try out some of the routines.

The tutorial ‘ZX Spectrum Machine Code Game in 30 Minutes!’ by Jon Kingsman looks intruiging, and promises the ability to program in machine code in the time it takes to consume a large cup of tea. I tend to ‘learn by doing’ so I think I may tackle this next.

Sources

• Assembly on the ZX Spectrum - Part 1, Benjamin Blundel
• Original source of this ‘Hello, world’ code may have been DIDIx13/hello-world
• Spectrum Machine Language for the Absolute Beginner, ed. William Tang. 1982 (2020 reprint)
• Jonathan Cauldwell, How to Write ZX Spectrum Games, v1.1 (?2020): ‘Chapter Eighteen - Making Games Load and Run Automatically’
• Spectrum memory map, Dean Belfield
• Pasmo docs
• ChatGPT (GPT-4)

Requirements

If you have not already done so, install the following tools:

1. Docker Desktop
2. Homebrew: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
3. Composer: brew install composer
4. Node/npm: brew install node
5. DDEV: brew tap drud/ddev && brew install ddev
6. mkcert: mkcert -install

Installation

I initially followed this Digital Ocean guide and distilled the key steps into the below sequence. (The Digital Ocean tutorial also covers Linux.)

1. mkdir <project-name>
2. cd <project-name>
3. ddev config --project-type=drupal9 --docroot=web --create-docroot
4. ddev start
5. ddev composer create "drupal/recommended-project"
   • This will install the latest stable version of Drupal. To choose an older version, say v10, run ddev composer create "drupal/recommended-project:^10"
6. ddev composer require "drush/drush"
   • If you get a PHP version error running this command:
     1. Update php_version in .ddev/config.yaml
     2. ddev restart
     3. Run step 6 again
7. ddev exec drush site:install --account-name=admin --account-pass=admin (or replace admin with a more secure username and password)
8. Modify the path for the config files in the DDEV settings file: sed -i '' "s|^# \\\$settings\['config_sync_directory'\].*|\$settings['config_sync_directory'] = '../config/sync';|" web/sites/default/settings.php
9. ddev launch

The new Drupal site, with the URL https://<project-name>.ddev.site/, should now open in your browser.

That’s it

You might like to install Bootstrap5 as a starter theme: composer require 'drupal/bootstrap5'. And these modules:

1. Simple XML sitemap: composer require drupal/simple_sitemap && ddev drush pm:enable simple_sitemap
2. Admin Toolbar: composer require drupal/admin_toolbar && ddev drush pm:enable admin_toolbar admin_toolbar_tools admin_toolbar_search
3. Pathauto: composer require drupal/pathauto && ddev drush pm:enable pathauto

At this point you might commit the code to a git repo, then get on with developing your site.

If you have any corrections or optimisations, please comment below.

For this blog post I’ve chosen a practical, real-life example from my archive of ChatGPT conversations. The problem in this case: I wanted to download a bunch of files (PDFs mostly) for which I had a list of URLs. My initial thought was to look into whether I could run something in the browser, for three reasons:

1. I’m a front-end dev whose natural habitat is the browser, so I should be able to whip up a solution quickly.
2. You don’t need to worry about authentication (if the files are only accessible when you’re logged in) as you can run the code from within an authenticated browser session.
3. The code can be run by others easily on their own machines.

On being presented with this problem, rather than start from a blank slate, I had a conversation with ChatGPT (GPT-4) about it. I’ve reproduced a lightly edited version of it below.

I’m aware that in blogging about a ChatGPT conversation I could be accused of being a bore (like somebody going on about the dream they had last night). However, large language model-based products, like ChatGPT and GitHub Copilot, look like they’re ushering in a major change in how sofware developers work, so I thought why not write about how I’m using them.

ChatGPT maintains context throughout the conversation so I can ask it questions about the information it’s given me (‘What does async do?’ and ‘Is that called “destructuring”?’), and ask it to refactor code to meet new requirements (‘Can you refactor downloadPDF() to handle multiple PDFs?’, ‘Can I determine the MIME type?’).

If you’re in a hurry, you can jump straight to the final iteration of the code.

JavaScript function to fetch and download PDF.

To fetch and download a PDF using JavaScript, you can use the fetch function to get the PDF data as a Blob and then create a download link using that Blob. Here’s an example function that takes a URL and a filename as parameters:

async function downloadPDF(url, filename) {
  try {
	// Fetch the PDF data as a Blob
	const response = await fetch(url);
	const blob = await response.blob();

	// Create a download link and set its attributes
	const link = document.createElement('a');
	link.href = URL.createObjectURL(blob);
	link.download = filename;

	// Append the link to the document, click it, and remove it
	document.body.appendChild(link);
	link.click();
	document.body.removeChild(link);
  } catch (error) {
	console.error('Error while downloading PDF:', error);
  }
}

// Usage example
const pdfURL = 'https://example.com/path/to/your/pdf-file.pdf';
const pdfFilename = 'downloaded-pdf-file.pdf';
downloadPDF(pdfURL, pdfFilename);

Make sure to replace the pdfURL variable with the URL of the PDF file you want to download. The pdfFilename variable is the name that will be given to the downloaded file.

[Tangential question removed.]

Re the downloadPDF() function you wrote, can you refactor to handle a list of PDFs to download, each with a specific, defined value[?]

Certainly! Here’s a modified version of the downloadPDF function that accepts an array of objects, each containing the URL and filename for a PDF. The function will download all PDFs one after another.

async function downloadPDFs(pdfs) {
  // [Deleted]

  for (const pdf of pdfs) {
	const { url, filename } = pdf;
	try {
	  // Fetch the PDF data as a Blob
	  const response = await fetch(url);
	  const blob = await response.blob();

	  // Create a download link and set its attributes
	  const link = document.createElement('a');
	  link.href = URL.createObjectURL(blob);
	  link.download = filename;

	  // Append the link to the document, click it, and remove it
	  document.body.appendChild(link);
	  link.click();
	  document.body.removeChild(link);

	  // Add a delay to avoid triggering browser's security mechanisms
	  await new Promise((resolve) => setTimeout(resolve, 1000));
	} catch (error) {
	  console.error(`Error while downloading ${filename}:`, error);
	}
  }
}

// Usage example
const pdfList = [
  {
	url: 'https://example.com/path/to/your/pdf-file1.pdf',
	filename: 'downloaded-pdf-file1.pdf',
  },
  {
	url: 'https://example.com/path/to/your/pdf-file2.pdf',
	filename: 'downloaded-pdf-file2.pdf',
  },
  // Add more PDFs as needed
];

downloadPDFs(pdfList);

Replace the pdfList array with the URLs and filenames for the PDF files you want to download. The function will iterate through the list and download each PDF file with the specified filename.

Please note that downloading multiple files in quick succession might be flagged by some browsers as suspicious behavior, potentially causing the downloads to be blocked. To avoid this, a delay is added between downloads using setTimeout. You can adjust the delay time as needed.

What does the async before the function definition mean?

The async keyword before a function definition indicates that the function is asynchronous. An asynchronous function is a function that can perform operations without blocking the execution of other code, allowing other tasks to run concurrently.

Asynchronous functions use the await keyword within their bodies to pause the execution of the function until a Promise is resolved or rejected. When a Promise is resolved, the function continues execution from the point where it was paused.

Here’s a simple example to illustrate how async functions work:

async function fetchData() {
  console.log('Fetching data...');

  // Simulate an asynchronous operation (e.g., fetching data from an API)
  const result = await new Promise((resolve) =>
	setTimeout(() => resolve('Data fetched!'), 1000)
  );

  console.log(result);
}

console.log('Starting...');
fetchData();
console.log('Waiting for data...');

The output will be:

Starting...
Fetching data...
Waiting for data...
Data fetched!

As you can see, the async function does not block the execution of the rest of the code. While it’s waiting for the Promise to resolve, other code (in this case, the console.log('Waiting for data...')) can run.

In the downloadPDFs function example, the async keyword allows the function to wait for the fetching and downloading of each PDF without blocking the execution of other code. This is useful when working with network requests or other asynchronous operations.

In the refactored downloadPDFs() function, what does const { url, filename } = pdf; do? Is that called ‘destructuring’ or something?

Yes, you’re correct! The line const { url, filename } = pdf; is an example of object destructuring in JavaScript. Object destructuring is a syntax feature that allows you to extract properties from an object and create new variables that hold the values of these properties.

In the downloadPDFs() function, the pdfs parameter is an array of objects, where each object has a url and filename property. The line const { url, filename } = pdf; is used to extract the url and filename properties from the current pdf object in the loop. This creates two new variables url and filename holding the respective values, which can then be used in the following lines of code.

Here’s a simple example to illustrate object destructuring:

const person = {
  name: 'John',
  age: 30,
};

const { name, age } = person;

console.log(name); // Output: 'John'
console.log(age);  // Output: 30

In this example, the name and age properties are extracted from the person object, and new variables name and age are created with the corresponding values.

Can you do this with arrays, for example if pdfList was an array of arrays, each PDF array being the format ['https://example.com/pdfs/pdf1.pdf', 'pdf1.pdf']?

Yes, you can use array destructuring to handle the pdfList if it’s an array of arrays. Array destructuring works similarly to object destructuring but is used to extract elements from arrays. Here’s the modified version of the downloadPDFs() function that works with an array of arrays:

async function downloadPDFs(pdfs) {
  for (const pdf of pdfs) {
	const [url, filename] = pdf; // Array destructuring
	try {
	  // Fetch the PDF data as a Blob
	  const response = await fetch(url);
	  const blob = await response.blob();

	  // Create a download link and set its attributes
	  const link = document.createElement('a');
	  link.href = URL.createObjectURL(blob);
	  link.download = filename;

	  // Append the link to the document, click it, and remove it
	  document.body.appendChild(link);
	  link.click();
	  document.body.removeChild(link);

	  // Add a delay to avoid triggering browser's security mechanisms
	  await new Promise((resolve) => setTimeout(resolve, 1000));
	} catch (error) {
	  console.error(`Error while downloading ${filename}:`, error);
	}
  }
}

// Usage example
const pdfList = [
  [
	'https://example.com/path/to/your/pdf-file1.pdf',
	'downloaded-pdf-file1.pdf',
  ],
  [
	'https://example.com/path/to/your/pdf-file2.pdf',
	'downloaded-pdf-file2.pdf',
  ],
  // Add more PDFs as needed
];

downloadPDFs(pdfList);

In this version, the line const [url, filename] = pdf; uses array destructuring to extract the URL and filename from each array element in pdfList. The rest of the function remains the same.

Say some of the files were Word docs (.docx); can I determine the MIME type from the response so I can give the file to be downloaded the correct file extension?

Yes, you can determine the MIME type of the fetched file from the Content-Type header in the response object. Based on the MIME type, you can set the appropriate file extension for the downloaded file. Here’s the modified version of the downloadPDFs() function that handles different file types and sets the correct file extension:

async function downloadFiles(files) {
  for (const file of files) {
	const [url, filename] = file;
	try {
	  // Fetch the file data as a Blob
	  const response = await fetch(url);
	  const blob = await response.blob();

	  // Get the MIME type from the response headers
	  const contentType = response.headers.get('content-type');
	  let extension;

	  // Determine the file extension based on the MIME type
	  switch (contentType) {
		case 'application/pdf':
		  extension = '.pdf';
		  break;
		case 'application/vnd.openxmlformats-officedocument.wordprocessingml.document':
		  extension = '.docx';
		  break;
		// Add more cases for other MIME types if needed
		default:
		  extension = ''; // Fallback to no extension if MIME type is unknown
	  }

	  // Create a download link and set its attributes
	  const link = document.createElement('a');
	  link.href = URL.createObjectURL(blob);
	  link.download = filename + extension;

	  // Append the link to the document, click it, and remove it
	  document.body.appendChild(link);
	  link.click();
	  document.body.removeChild(link);

	  // Add a delay to avoid triggering browser's security mechanisms
	  await new Promise((resolve) => setTimeout(resolve, 1000));
	} catch (error) {
	  console.error(`Error while downloading ${filename}:`, error);
	}
  }
}

// Usage example
const fileList = [
  [
	'https://example.com/path/to/your/pdf-file1.pdf',
	'downloaded-pdf-file1',
  ],
  [
	'https://example.com/path/to/your/word-file1.docx',
	'downloaded-word-file1',
  ],
  // Add more files as needed
];

downloadFiles(fileList);

In this version, the function is renamed to downloadFiles() to reflect that it can handle different file types. The MIME type is obtained from the response headers using response.headers.get('content-type'), and a switch statement is used to determine the appropriate file extension based on the MIME type. The link.download attribute is set to the filename concatenated with the determined file extension.

Please note that this approach relies on the server providing the correct Content-Type header for the files. If the server doesn’t provide the correct MIME type, you might need to use an alternative method, such as checking the file extension in the URL or using a library to determine the file type based on the content.

And that was me with the code I needed. I manually tidied it up a bit, and this is the format of the final version that I ran:

async function downloadFiles(files) {

  // MIME type: file extension
  const fileExtensions = {
	'application/pdf': 'pdf',
	'application/vnd.openxmlformats-officedocument.wordprocessingml.document': 'docx',
	'application/msword': 'doc',
	'application/zip': 'zip'
  }

  const mimeTypes = [];

  // Collate MIME types
  Object.keys(fileExtensions).forEach(key => {
	mimeTypes.push(key);
  });

  for (const file of files) {
	const [url, filename] = file;

	try { // Fetch the file data as a Blob

	  const response = await fetch(url);
	  const contentType = response.headers.get('Content-Type');

	  if (response.ok && mimeTypes.includes(contentType)) {
		const blob = await response.blob();

		// Create a download link and set its attributes
		const link = document.createElement('a');
		link.href = URL.createObjectURL(blob);
		link.download = `${filename}.${fileExtensions[contentType]}`;

		// Append the link to the document, click it, and remove it
		document.body.appendChild(link);
		link.click();
		document.body.removeChild(link);
	  } else {
		console.error(`Error while downloading ${filename}: Invalid content type (${response.headers.get('Content-Type')}) or response error`);
	  }

	  // Add a delay to avoid triggering browser's security mechanisms
	  await new Promise((resolve) => setTimeout(resolve, 300));
	} catch (error) {
	  console.error(`Error while downloading ${filename}:`, error);
	}
  }
}

// Usage example
var fileList = [ // `const` → `var` so we run in single browser console session more than once
  ['https://example.com/path/to/your/pdf-file1.pdf', 'pdf-file1'],
  ['https://example.com/path/to/your/word-file1.docx', 'word-file1'],
];

downloadFiles(fileList);

This was a pretty straightforward series of prompts. The problem was a relatively simple one, and ChatGPT was able to output what I wanted from start to finish. In more complex scenarios, it might only get you started, or you’ll have enough code you want to ask it about that you’ll run out of tokens. ChatGPT also didn’t have any hallucinations in this session. When that happens you can politely let it know it’s havering and should provide a new answer.

And as to the way I went about solving this problem (running a script from the browser console), some may think it a bit inelegant or hacky. You’re probably right! But it did the job and that’s what matters. And thanks to ChatGPT it took less time and effort than it would have otherwise.

Disclaimer: I’m not a Drupal expert and share this post only to help others who may have a similar use case. I make no guarantees of best practice! This should work in Drupal 8 onwards. Refinements welcome in the comments.

The scenario here is that you want to create a number of landing pages which, within the body of each landing page, list their immediate sub-pages. Along with a hyperlink, you also want to output a thumbnail image which has been assigned to each sub-page.

Say you were building a website for a professional wrestling organisation called the Scottish Wrestling Federation (SWF) and had a number of categories under a parent Roster page: Men, Women, Broadcast team and Referees. Each landing page is a gallery of photos which link to a relevant profile page.

First create a custom Image field called field_wrestler_photo (most of the roster are wrestlers so we’ll ignore the fact that a few are referees and announcers in terms of the naming). Then set up a Menu block called Sub-pages within the Main content region at /admin/structure/block. You should now see a theme hook suggestion at /admin/structure/block/manage/sub_pages#edit-settings-style (within the HTML and style options settings). Let’s name the block menu__sub_pages.

Next add a function called swf_preprocess_menu__sub_pages() to our swf.theme file. This function will process the URL and alt text data and assign it to properties of $variables. The data will then be available to us in menu--sub_pages.html.twig in the templates/navigation folder.

We’ll be using the Node interface within our preprocess function in swf.theme, so first include it towards the top of the file:

use Drupal\node\Entity\Node;

Begin our function by setting up a loop to get the data for each page:

function swf_preprocess_menu__sub_pages(&$variables) {
  foreach ($variables['items'] as $key => $item) { // Loop through each page
    // ...
  }
}

For each page, get its URL (eg /roster/women/jean-meikle) and node path (eg /node/23). From the node path use preg_match() to get the node ID (23 in this example):

function swf_preprocess_menu__sub_pages(&$variables) {
  foreach ($variables['items'] as $key => $item) {
    $page_url = $variables['items'][$key]['url']->toString(); // Get page’s URL
    $node_path = \Drupal::service('path_alias.manager')->getPathByAlias($page_url); // Get node path, eg /node/23
    if (preg_match('/node\/(\d+)/', $node_path, $matches)) {
      $node_id = intval($matches[1]); // Get the numeric node ID
      // ...
    }
  }
}

Next, check for the property field_wrestler_photo and whether it has a value. Then get the Node object for this node ID:

function swf_preprocess_menu__sub_pages(&$variables) {
  foreach ($variables['items'] as $key => $item) {
    $page_url = $variables['items'][$key]['url']->toString();
    $node_path = \Drupal::service('path_alias.manager')->getPathByAlias($page_url);
    if (preg_match('/node\/(\d+)/', $node_path, $matches)) {
      $node_id = intval($matches[1]);

      if (
          $node_id &&
          Node::load($node_id)->field_wrestler_photo && // Has wrestler photo field
          count(Node::load($node_id)->field_wrestler_photo) // Field has value
      ) {
        $node = Node::load($node_id); // Get node object
        // ...
      }
    }
  }
}

You can now access the alt text, and use getFileUri() and the file_url_generator service to get the image’s URL. Finally, assign them to $variable properties so you can reference them in the Twig templates:

function swf_preprocess_menu__sub_pages(&$variables) {
  foreach ($variables['items'] as $key => $item) {
    $page_url = $variables['items'][$key]['url']->toString();
    $node_path = \Drupal::service('path_alias.manager')->getPathByAlias($page_url);
    if (preg_match('/node\/(\d+)/', $node_path, $matches)) {
      $node_id = intval($matches[1]);

      if (
          $node_id &&
          Node::load($node_id)->field_wrestler_photo &&
          count(Node::load($node_id)->field_wrestler_photo)
      ) {
        $node = Node::load($node_id);
        $image_alt_text = $node->field_wrestler_photo->alt; // Get alt text
        $file_uri = $node->field_wrestler_photo->entity->getFileUri(); // Get file URI
        $image_url = \Drupal::service('file_url_generator')->generateString($file_uri); // Get image URL
        $variables['items'][$key]['image_alt_text'] = $image_alt_text; // Store alt text
        $variables['items'][$key]['image_url'] = $image_url; // Store image URL
      }
    }
  }
}

Create a Twig template for the roster lists in templates/navigation/menu--sub_pages.html.twig:

{% import _self as menus %}

{{ menus.build_menu(items, attributes, 0) }}

{% macro build_menu(items, attributes, menu_level) %}
  {% import _self as menus %}
  {% if items %}
  {% for item in items %}
    {{ menus.add_link(item, attributes, menu_level) }}
  {% endfor %}
  {% endif %}
{% endmacro %}

{% macro add_link(item, attributes, menu_level) %}
  <div>
    <a href="{{ item.url }}">
     <img src="{{ item.image_url }}" alt="{{ item.image_alt_text }}" width="240" height="360"> {# Wrestler image #}
    </a>
    <div>{{ item.title }}</div> {# Wrestler name #}
  </div>
{% endmacro %}

You can now create a page for each person in the roster and arrange them into categories by editing the menu struture at /admin/structure/menu/manage/main.

You probably want to restrict the sub-pages block to only appear on the Roster pages by adding relevant paths at /admin/structure/block/manage/sub_pages#edit-visibility-request-path: /roster, roster/men etc.

We’ve focused here on a custom image field, but you can access any default or custom field in a similar fashion. For example, you could add field_wrestler_height and field_wrestler_weight fields and list each wrestler’s height and weight under their photo.

Update (18 Feb): I’ve added a note about how ChatGPT works, and which may explain some of the nonsense it can produce.

It’s not hyperbole to say that ChatGPT is revolutionary. It’s a significant leap over the often frustrating experience of using Siri or Alexa.

Like Siri and Alexa, ChatGPT doesn’t always get everything right. It can state nonsense confidently as if it were fact. However, whether it’s searching the web, reading a newspaper article, or even an academic text, I’m quite used to having to have my bullshit detector switched on and comparing several sources to verify things. In that respect, ChatGPT seems as good as any other information source. (It got almost everything about my areas of expertise — the Scots language, software development and music — correct.)

Since its launch late last year I’ve been thinking about whether ChatGPT might free white-collar workers (or middle-class professionals, as you might call them here in the UK) from everyday office drudgery. We’ve been getting there over time: my first office job 20 years ago, where I transferred information from scanned paper copies of life insurance applications into text fields in a desktop software application by typing a letter at a time, surely no longer exists.

Could ChatGPT be a revolutionary, as opposed to evolutionary, step in freeing us from monotonous work (not before UBI though, please; I have a mortgage to pay)? I have questions about what AI tools might mean for content producers, office workers and even the future of society. I decided to put some of them to ChatGPT directly to see what it had to say. It was an informative chat. A transcript is below. The original text can be found on the OpenAI website.

Will ChatGPT ever list its sources so users can verify the information and do further reading?

ChatGPT is trained on a diverse range of internet text, which means it can generate responses based on the information it has seen during training. However, it does not store or have a way to retrieve the specific source of the information it provides. While it is not possible for ChatGPT to list its sources for users to verify information or do further reading, it is possible to develop a system that would keep track of the sources used to train a certain model version, so that users could identify potential sources of the information generated by the model. This can be done by keeping a record of the data used to train the model, but this would require significant computational resources and would make the model less efficient.

It‘s important to note that GPT-3 and other models like it should be considered as an aid and not a replacement for human knowledge and critical thinking. They should not be used as a sole source of information and their outputs should always be critically evaluated.

Will providers of the information used by ChatGPT be compensated in some way? For example, will a[n] academic receive any royalties if several of their papers contributed to the model?

It is unlikely that individual authors or sources of the text used to train ChatGPT or other language models will be compensated directly. The vast majority of the text used to train these models is taken from publicly available sources on the internet, such as websites and social media, which are generally not covered by copyright. Additionally, it would be extremely difficult to track down and compensate all of the individual authors and sources of the text used to train a model, especially given the sheer volume of text that is used.

In the case of academic papers, it‘s possible that the authors of these papers would not receive any royalties or compensation from the use of their papers in the training data set. Many academic papers are made available online for free and the authors have likely signed over the copyright to the publisher.

It‘s worth noting that OpenAI, the company behind ChatGPT, has a policy of only using text that is publicly available and not covered by copyright. Additionally, OpenAI has also committed to using a portion of its revenue to fund research and development in AI safety and open AI.

It‘s important to note that the use of text from any sources, whether they are publicly available or not, should always be done in compliance with the terms of use and copyright laws.

Could ChatGPT ingest highly technical sources that cover a particular domain, for example aerospace engineering, to create a custom model for use within a particular industry?

Yes, it is possible to fine-tune ChatGPT or other language models on a specific domain or industry, such as aerospace engineering. This process is called “domain adaptation” or “domain transfer.” By fine-tuning the model on a specific dataset of domain-specific text, the model can learn the specific terminology, concepts and writing style of that domain, which allows it to generate more accurate and relevant responses when working within that domain.

Fine-tuning can be done by using a dataset of technical documents, articles or papers that cover the specific domain such as aerospace engineering. The process involves training the model on this specific dataset, which allows it to adapt to the specific characteristics of the domain.

However, it‘s worth noting that fine-tuning a model on a specific domain requires a large amount of high-quality, domain-specific data and computational resources. Additionally, the fine-tuned model will perform best on the specific task and domain it was trained on and may not generalize well to other tasks or domains.

It‘s also important to keep in mind that a fine-tuned model may not perform as well as an expert in the field, and the outputs should be always evaluated critically.

Can you give a ballpark figure as to how much data would be required? For example, would 250,000 words be enough?

The amount of data required to fine-tune a language model on a specific domain depends on various factors such as the complexity of the domain, the quality of the data, and the size of the pre-trained model.

In general, the more data and the more diverse the data, the better the fine-tuned model will perform. However, as a rough estimate, around 250,000 words may be enough to fine-tune a model on a specific domain, but it‘s also possible that more or less data may be needed depending on the specific situation.

For example, for a specific task or domain like aerospace engineering, 250,000 words of high-quality, domain-specific text such as academic papers, technical reports and journal articles would likely be sufficient to fine-tune a large pre-trained language model like GPT-3. However, for a more complex and broader domain like medicine, it may require more data to achieve similar results.

It‘s worth noting that in practice, obtaining a large amount of high-quality, domain-specific data can be a challenging task and requires a considerable effort. Additionally, it‘s also important to keep in mind that fine-tuning a model on a specific domain requires computational resources and time, and the performance may vary depending on the specific task and data.

Can ChatGPT understand diagrams as well as standard text/prose?

ChatGPT is a language model, which means it is trained to understand and generate text. It is not specifically designed to understand diagrams or images, however, it can be fine-tuned on a dataset that includes images and captions, allowing the model to learn some level of understanding of image-text associations.

There are other models like image captioning models, which can be used in conjunction with language models like ChatGPT to understand and generate captions for images. These models are trained to understand the content of images and generate text descriptions of them.

It‘s worth noting that current state-of-the-art models for image captioning still have limited understanding of complex images and there is still a lot of research being done in this field to improve their ability to understand and generate captions for images.

Moreover, it‘s also important to keep in mind that a model‘s understanding of images is limited to the data it has seen during training, and it may not generalize well to new images or situations.

If ChatGPT doesn‘t know the answer to a user query, will it just say so? Would it ever give out a false answer if it didn’t know the correct answer?

ChatGPT is a machine learning model, which means that it is able to generate responses based on the patterns it has learned during training. However, it is not infallible and may not know the answer to every question or provide a false answer.

If ChatGPT doesn‘t know the answer to a user‘s query, it will likely generate a response that indicates that it doesn’t have enough information to provide an answer. For example, it may say something like “I‘m sorry, I don‘t know the answer to that question” or “I don‘t have enough information to provide an accurate answer.”

It‘s also worth noting that ChatGPT is trained on a vast amount of text data from the internet, and the quality of the information it has seen during training may vary. Therefore, it‘s possible that the model may generate a response that is incorrect, out-of-date or biased.

It is important to keep in mind that the output of a language model should be critically evaluated, and the model should not be considered as a sole source of information or knowledge.

OpenAI, the company behind ChatGPT, has an API which allows developers to access the model‘s capabilities and integrate them into their own applications, and they also provide a way to flag unsafe or unreliable content, which OpenAI team will review and take action if necessary.

Note (18 Feb)

I’ve experienced ChatGPT, as it describes above, saying it cannot give an answer (in response to being asked to translate Scots text of more than a sentence or two), but I’ve also seen it return utter falsehoods, where it seems like it doesn’t know the answer and is just making things up.

This makes some sense given that ChatGPT is, as described by Stephen McAleese at LessWrong, a ‘next-word’ predictor that ‘developed unexpected capabilities such as the ability to learn on the fly from prompts (in-context learning), arithmetic, and code generation.’ Stephen Wolfram describes ChatGPT’s approach to writing an essay as ‘essentially […] just asking over and over again “given the text so far, what should the next word be?”—and each time adding a word.’

How do you think ChatGPT will impact software development?

ChatGPT and other large language models like it have the potential to greatly impact software development in several ways:

1. Automated content generation: ChatGPT and other language models can be used to generate large amounts of high-quality text, such as documentation, product descriptions, and even code. This can greatly reduce the time and effort required for content creation and allow developers to focus on more complex tasks.
2. Improved natural language processing: ChatGPT and other language models can be used to improve natural language processing (NLP) capabilities in a wide range of applications, such as chatbots, virtual assistants, and sentiment analysis. This can make it easier for developers to create NLP-enabled applications that are more accurate and user-friendly.
3. Improved search: ChatGPT and other language models can be used to improve the quality of search results by better understanding the intent behind user queries. This can make it easier for developers to create search applications that are more accurate and relevant.
4. Improved data analysis: ChatGPT and other language models can be used to analyze and extract insights from large amounts of text data. This can make it easier for developers to create data-driven applications that are more accurate and informative.
5. Improved accessibility: ChatGPT and other language models can be used to improve accessibility for people with disabilities, such as those who are visually impaired or have difficulty reading. This can make it easier for developers to create applications that are more inclusive and accessible to a wider range of users.

Overall, ChatGPT and other large language models have the potential to greatly accelerate the development of a wide range of applications and services that rely on natural language processing and understanding. However, it‘s worth noting that the use of these models requires significant computational resources, and the development of applications that rely on these models may require a significant investment of time and resources.

How might ChatGPT improve accessibility for people with disabilities?

ChatGPT and other large language models have the potential to improve accessibility for people with disabilities in several ways:

1. Text-to-speech: ChatGPT and other language models can be used to generate high-quality text-to-speech (TTS) output, which can make it easier for people with visual impairments to access and understand written content.
2. Automatic captioning: ChatGPT and other language models can be used to generate automatic captions for videos, which can make it easier for people with hearing impairments to access and understand spoken content.
3. Language translation: ChatGPT and other language models can be used to translate text from one language to another, which can make it easier for people with language barriers to access and understand written content.
4. Speech recognition: ChatGPT and other language models can be used to improve speech recognition capabilities in a wide range of applications, such as virtual assistants and chatbots, which can make it easier for people with mobility impairments or difficulty typing to interact with technology.
5. Summarization: ChatGPT and other language models can be used to summarize long text content, which can make it easier for people with cognitive impairments or difficulty reading to access and understand written content.

It‘s worth noting that the use of these models to improve accessibility for people with disabilities is still an active research area, and there are still some challenges to be overcome, such as the need for large amounts of high-quality, diverse data and the ability to fine-tune the models to specific use cases and users. Additionally, the performance of these models may vary depending on the specific task and data.

Can ChatGPT deal with languages that have some variation in spelling, for example, Scots and Nynorsk? I recently asked you about the meaning of a few Scots words and phrases and you did well despite the spelling variation.

Yes, ChatGPT and other large language models like it can deal with languages that have some variation in spelling, such as Scots and Nynorsk. However, it‘s important to note that the performance of the model will depend on the amount and quality of data it has seen during training.

Large language models like ChatGPT are trained on vast amounts of text data from the internet, which means that they are exposed to a wide range of variations in spelling, grammar, and language usage. This allows the model to learn the different ways that words and phrases can be spelled and used, and thus to generalize well to new, unseen variations.

However, it‘s worth noting that the model‘s performance may not be perfect, especially if it has not seen enough examples of a specific variation during training. This is particularly true for low-resource languages or dialects, which may have less representation in the training data.

In the case of Scots, a dialect spoken in Lowland Scotland, it is similar to English, but with some variations in spelling, grammar, and vocabulary. However, the model can still understand and generate text in this dialect, thanks to the large amount of training data it has seen.

It‘s also important to keep in mind that the model‘s understanding of any language is limited to the data it has seen during training, and it may not generalize well to new, unseen variations of that language or dialect, especially if they are not well represented in the training data.

Thanks. Just to note: Scots is recognised as a language, rather than a dialect, by many. Of course, dialect and language can be used interchangeably, and Scots and English are closely related, so I won‘t take offense!

I apologize for any confusion caused, you are correct that Scots is recognized as a language by many, and it has its own distinct spelling, grammar, and vocabulary. I apologize for any confusion caused by my previous message. The distinction between dialect and language can be complex, and different people may have different opinions on how to classify different forms of speech.

As a machine learning model, ChatGPT‘s understanding and classification of languages is based on the data it has seen during training. If the training data includes examples of Scots as a language, the model will be able to understand and generate text in this language.

It‘s important to remember that a machine learning model‘s understanding of a language will always be limited by the data it has seen during training, and it may not fully understand or reflect the nuances and complexities of a language, dialect or culture.

Does ChatGPT attach some kind of metric or parameter to each answer which estimates exactly how accurate the response likely is?

ChatGPT, like most language models, does not attach a metric or parameter to each answer that estimates how accurate the response likely is. Instead, it generates responses based on the patterns it has learned during training, and the quality of the responses can vary depending on the input and context.

Language models like ChatGPT are typically trained on a large amount of text data from the internet, and the quality of the information in this data can vary. Therefore, while the model may generate a response that is accurate and relevant in some cases, it may generate a response that is inaccurate, out-of-date, or biased in other cases.

There are some ways to evaluate the quality of the responses generated by a language model, such as human evaluation, where a human annotator assess the quality of the response, or automatic evaluation by comparing the generated response against a reference set. However, these methods are not always straightforward, and the results may vary depending on the specific task and data.

Additionally, it‘s important to remember that the output of a language model should always be critically evaluated, and the model should not be considered as a sole source of information or knowledge.

Could ChatGPT ever replace Google as the main way people search for information?

It is unlikely that ChatGPT or any other single language model could replace Google as the main way people search for information. Google Search is a complex system that uses a variety of techniques, such as web crawling, indexing, and ranking, to provide relevant and accurate search results to users.

ChatGPT and other language models like it are primarily focused on understanding and generating text, rather than searching and indexing the web. While they can be used to improve the quality of search results by better understanding the intent behind user queries, they are not currently capable of indexing and ranking the entire web.

Additionally, Google search not only returns the most relevant results but also provides a vast array of features like autocomplete, spell check, voice search and many more.

It‘s worth noting that OpenAI, the company behind ChatGPT, has developed a search engine called DALL-E that can search and generate images, but it‘s still in its early stages and not available for general use yet.¹

That being said, ChatGPT and other language models like it can be used to improve the natural language understanding capabilities of search engines and other applications, making them more accurate and user-friendly. But it‘s unlikely that it will replace Google or other search engines as the main way people search for information in the near future.

Do you expect ChatGPT will put a lot of white[-]collar workers out of work?

ChatGPT and other large language models like it have the potential to automate certain tasks that were previously done by white-collar workers, such as content generation, data entry, and customer service. However, it‘s important to note that the impact of these models on employment will depend on a variety of factors, including the specific tasks and industries in question, as well as the rate at which the technology improves and becomes more widely adopted.

It‘s likely that the use of ChatGPT and other large language models will result in some job displacement, particularly in areas such as data entry, customer service and content creation. However, it‘s also possible that the use of these models will lead to the creation of new types of jobs and industries, such as those related to building, training, and maintaining the models.

It‘s also worth noting that the use of language models like ChatGPT can also bring benefits to businesses and society as a whole by increasing efficiency and productivity, reducing costs, and improving the quality of products and services.

It‘s important to keep in mind that the impact of ChatGPT and other language models on employment is a complex issue that depends on many factors. It‘s also important to consider the potential benefits and drawbacks of the technology, and to actively work on the development of policies and programs that can mitigate the negative effects and maximize the positive effects of the technology on employment and society.

It is possible, on an infinite timescale, that ChatGPT or another AI could actually become sentient?

The question of whether an AI could become sentient is a topic of ongoing debate and research in the field of artificial intelligence. Sentience is the capacity to have subjective experiences and conscious awareness, and it‘s a difficult concept to define and measure.

Currently, the AI systems like ChatGPT and other large language models are based on machine learning techniques that are used to process and analyze large amounts of data, but they don‘t have the ability to feel emotions, consciousness, self-awareness or experience the subjective aspects of the world.

It‘s important to note that the current AI systems are not sentient, they are based on mathematical models and algorithms, and their behavior is determined by the parameters and data that is used during training. There‘s no evidence that current AI systems have the ability to become sentient, and it‘s not clear if it would ever be possible for an AI to become sentient or not.

That being said, the field of AI is rapidly evolving, and new developments in the field of Artificial General Intelligence (AGI) and Artificial Super Intelligence (ASI) may bring new possibilities. However, the question of whether an AI could become sentient is still a topic of debate and it‘s not something that is likely to happen in the near future.

That‘s somewhat of a relief.

I understand that the idea of an AI becoming sentient can be a concern for some people. It‘s important to remember that the current state of AI technology is still quite limited, and the capabilities of AI systems like ChatGPT are still far from matching human intelligence.

It‘s also worth noting that the development of AI is guided by a set of ethical principles, such as transparency, accountability and explainability, which aim to ensure that the technology is used for the benefit of humanity and that its development is guided by ethical considerations.

It‘s important to continue to have open and honest conversations about the potential implications of AI and to actively work on the development of policies and programs that can mitigate the negative effects and maximize the positive effects of the technology on society.

Thank you for answering my questions.

You‘re welcome! I‘m glad I could help. If you have any other questions, feel free to ask.

When coming up with a domain name for your latest venture, the possibilties now seem endless. Even if a .com is still available,² there’s probably something more eye-catching and a better fit for your project out there.

gTLDs can have their own tone, ranging from the irreverent (.fail and .lol) to the functional (.plumbing and .florist) to the apparently inexplicable (.ooo and .xyz), and they seem to have become somewhat more visiable of late. For example, the advice columnist and podcast host Dan Savage’s domain is now savage.love,³ and I’ve just seen freetesting.hiv in an Instagram ad.⁴ The former GoCompare.com (quite well known in the UK owing to their annoying TV ads) has gone for a full rebrand as Go.Compare, with the idea being that ‘searching’ for it on any device will take the user directly to the website rather than an intermediary search engine.

The project I’ve been working on for the past couple of years is a website with a bunch of jazz piano tips and tricks. In the process of thinking about what I should call it, I jotted down some ideas. I thought I’d publish them here as a blog post, along with comments on their suitability. I’m not sure whether the list provides much in the way of insight into how to best go about choosing the ‘perfect’ domain name, but I present it here nonetheless for anyone looking for inspiration.

(These are the domains in the order I jotted them down in my notes file. You can mostly interchange ‘jazz’, ‘jazzkeys’ and ‘jazztoolkit’, as it took some time before I settled on which variant I liked best.)

• jazz.wtf
  I like this. It’s irreverent. Reflects the state of feeling overwhelmed when learning a complex subject like jazz. Maybe a bit sweary for some people, though.
• jazz.clinic
  Nice. It might suggest that the website is some kind of a workshop, which it’s not. Maybe a bit too grand also: I don’t feel in a position to put on a ‘clinic’ in jazz piano. And aren’t clinics more things that drummers do?
• jazz.tips
  Like it. Short. Doesn’t suggest something too all-encompassing.
• jazztoolkit.fyi
  I like the informality of the ‘.fyi’ domain. Reflects the approach I’m taking. ‘Toolkit’ feels like it reflects what the site will be: a bunch of techniques and approaches to playing jazz piano.
• jazzpiano.how
  Cool domain, but .how suggests too grand a scope. The site will be a collection of tips, examples etc.; not a soup-to-nuts course in how to play jazz piano. I’m not sure even the most accomplished jazz piano player would be so arrogant as to think they know the ‘how’: making music is an individual, and often mysterious, thing.
• jazz.fyi
  Nice. However, I’m leaning towards focusing on just jazz piano and not jazz in general (though a lot of stuff can also be applied to other instruments). Domain is already registered so I’d need to look into making an offer.
• jazz.fail
  Like jazz.wtf, this is irreverent, and I like it; but maybe a bit negative sounding.
• jazzpiano.tools
  Goes well with the ‘toolkit’ framing.
• jazzkeys.tools
  Fine. I prefer ‘keys’ to ’piano’. It’s more informal, and I’m recording a lot of the examples on an electric piano anyway.
• jazztoolkit.online
  OK, I guess. There are better TLDs than .online.
• jazztoolkit.help
  Cool TLD that reflects what I want to achieve with the site.
• jazztoolk.it
  Too clever. Difficult to verbally convey to others. Not a huge fan of using country-level TLDs for purposes other than those for which they’re intended.
• jazz.help
  Fine, but at ~£100 it’s a bit expensive.⁵
• jazztoolkit.app
  I like the .app TLD, and the site does have app-like functionality; but it might confuse non-tech people as it’s a web app, which I don’t think means much to non-tech punters.
• 🌟 jazzkeys.fyi
  OK, this is perfect. Suggests an informal, non-prescriptivist approach: “Here’s what works for me; if it’s of use to you, then great.” Also rolls off the tongue. JazzKeys.fyi it is!

JazzKeys.fyi will be launching soon. Sign up to be notified!

tl;dr: if you’d rather just dig into the code, and not bother with this blog post, there’s a repo on GitHub.

This post describes a demo version — available at the repo linked above — of the code I deployed on my home page.

Contents

• Testing the API
• Authenticating
• Proxy scripts
  • header.inc
  • fetch.php: fetch a listing of gists for a user
  • fetch-id.php: fetch content of a specific gist
  • Cross-Origin Resource Sharing (CORS)
  • Folder structure
• Building the front end
  • Installing a custom build of highlight.js as a npm dependency
  • index.html
  • index.js
• Summary

Testing the API

You can query the API to fetch a list of gists for a specific user without authenticating. For example, running this code in your browser console will return a listing of my 15 most recent gists in JSON format (optionally replace donbrae with your own user name):

fetch('https://api.github.com/users/donbrae/gists?per_page=15').then(response => {

  // Success
  if (response.ok)
    return response.json(); // Returns to then()

  // Error
  return Promise.reject(response);

}).then(data => {
  console.log(data);
}).catch(err => {
  console.error(err); // Error
});

You can also fetch the actual content of a gist by using the /gists/{gist_id} endpoint, eg https://api.github.com/gists/d78ee08d2ffdc2f7b8442155f9cf7fa1.

Authenticating

For unauthenticted users, the API is rate-limited to 60 requests per hour. Creating a personal access token¹ and using it to authenticate with the API gets you 5,000 an hour.

When authenticating, we don’t want to expose the access token by making the request directly on the client side, so we can set up proxy scripts on a server to do the authenticating and make the API calls.

I wrote the scripts in PHP because I have some familiarity with the language, and it comes installed with your typical web hosting plan, but you can translate it to your preferred language.

Proxy scripts

The two proxy scripts — one to fetch the list of gists, and one to fetch a particular gist’s content — are in the repo donbrae/get-gists.

header.inc

header.inc contains code we’ll use in both scripts. First we add headers to set the Content-Type to application/json (we’ll be serving JSON), and the Cache-Control header to no-cache (we’ll be implementing our own caching system).

<?php
header('Content-Type: application/json');
header('Cache-Control: no-cache');

Next, the variable $opts contains our HTTP context options, including the request header values we’ll pass to the GitHub API. Replace <token> with your personal access token.

$opts = [
  'http' => [
    'method' => 'GET',
    'header' => [
      'User-Agent: PHP',
      'Content-Type: application/json',
      'Accept: application/vnd.github+json',
      'Authorization: token <token>'
    ]
  ]
];

There is also a function that writes files in a way that avoids potential write conflicts:

function writeFile($filename, $content, $append = false) {
  // https://softwareengineering.stackexchange.com/a/332544
  $unique_tmp_filename = uniqid('', true) . '.tmp'; // Create unique filename
  if ($append) {
    file_put_contents($unique_tmp_filename, file_get_contents($filename) . $content); // Concatenate contents of existing file with new content
  } else {
    file_put_contents($unique_tmp_filename, $content);
  }

  rename($unique_tmp_filename, $filename);
}

And a function to parse the response headers:

function getResponseHeaders($http_response) {
  // https://beamtic.com/parsing-http-response-headers-php
  $response_headers = [];
  foreach ($http_response as $value) {
    if (false !== ($matches = explode(':', $value, 2))) {
      $response_headers["{$matches[0]}"] = trim($matches[1]);
    }
  }
  return $response_headers;
}

And, finally, a function to get the HTTP response status code as an integer:

function getStatus($http_response) {
  // https://stackoverflow.com/a/52662522/4667710
  $status_message = array_shift($http_response);
  preg_match('{HTTP\/\S*\s(\d{3})}', $status_message, $match);
  $status = intval($match[1]);
  return $status;
}
?>

(You probably won’t need the last two functions if you’re not using PHP. PHP is weird.)

fetch.php: fetch a listing of gists for a user

Now onto the script that will actually get the data, fetch.php.

We start by including the header.php file:

<?php
require_once 'header.inc';

Next, name a file that we’ll use as a cache:

$cache_file = 'cached.json';

Now we’ll call the API and store the returned data in variable $content:

$context = stream_context_create($opts);
$content = file_get_contents('https://api.github.com/users/donbrae/gists?per_page=15', false, $context);

We’ll then use a couple of the other functions we declared in header.php to parse out the response status code and headers. This data is available in the $http_response_header array:

$status = getStatus($http_response_header);
$response_headers = getResponseHeaders($http_response_header);

If the status is 200, we output the JSON returned from the API:

if ($status === 200) {
  echo $content;

Next we add a conditional that checks the X-RateLimit-Remaining response header to determine whether we should write a copy of the returned JSON to our cache file:

  if (
    intval($response_headers['X-RateLimit-Remaining']) < 250 && // We may hit the API rate limit soon
    file_exists($cache_file) && (time() - filemtime($cache_file)) / 60 / 60 > 6 || // Cached file hasn’t been updated in last 6 hours
    !file_exists($cache_file) // Or there is no cache file
  ) {
    writeFile($cache_file, $content); // Write a cached version of JSON in case API rate limit is reached
  }

I set the threshold to an arbitrary 250 API calls remaning. The conditional could instead in theory be X-RateLimit-Remaining === 0, but the script that fetches the actual gist content (fetch-id.php) also counts towards API usage, so we can’t guarantee that fetch.php will be called when the rate limit is exactly zero.²

There is also a condition which checks that the cache file hasn’t been updated in the last six hours, to avoid continually writing a new cache file every time this script is called and X-RateLimit-Remaining is under 250.

Alternatively, if no cache file exists — regardless of other conditions — write one.

If another status is returned, and if there is a cached version, serve the cached version; otherwise, return an error message in JSON format:

} else if (file_exists($cache_file)) { // Status ?304, 403 or 422
  echo file_get_contents("cached.json"); // Serve cached copy
} else {
  echo "{\"error\": \"Cannot fetch list of gists. Error code: $status\"}";
}
?>

CORS

Add fetch.php to your server in a folder called gists. If you’re using another domain to host the proxy scripts — as I’m doing because my blog is a static site hosted on Netlify, which doesn’t support PHP — we’ll need to make sure Cross-Origin Resource Sharing (CORS) is set up properly. We need to tell our server hosting fetch.php to accept Ajax requests from our originating domain. In my case, the proxy domain — donbrae.co.uk — needs to accept requests from browsers viewing jamieonkeys.dev.

To do this — and assuming your server runs Apache — add a .htaccess file in the gists folder on donbrae.co.uk:

<IfModule mod_headers.c>
  Header set Access-Control-Allow-Origin: https://www.jamieonkeys.dev
</IfModule>

The website at jamieonkeys.dev should now be able to call https://donbrae.co.uk/proxy-php/gists/fetch.php and get the data in return.

fetch-id.php: fetch content of a specific gist

There is also a proxy script, fetch-id.php, to get the content of specific gists.

Again, we start by including the same header.inc file we used previously:

<?php
require_once 'header.inc';

On the front end we’ll be passing a gist ID in the querystring parameter gist_id, so capture that:

$gist_id = $_GET['gist_id'];

The GitHub API returns a status code of 304 for gists which haven’t been updated since a specified date. Requests which result in a 304 also don’t count towards your rate limit. We’ll create a caching system to return static JSON files from our server so we don’t need to keep calling the API.

Add a subfolder called on your server under gists called gistscache.

fetch-id.php continues:

$cached_file_path = "./gistscache/$gist_id.json";
$cached_file_exists = file_exists($cached_file_path);

if ($cached_file_exists) {
  // Get last updated date of file in GMT
  $last_modified = filemtime($cached_file_path); // Get its last modified date
  $last_modified_gmt = gmdate('D, d M Y H:i:s', $last_modified) . ' GMT'; // Format it so GitHub API accepts it in header

  // Add date to header so that API will return 304 if no updates have been made, and we can serve the cached copy; otherwise we can expect a 200 with the latest data
  array_push($opts['http']['header'], "If-Modified-Since: $last_modified_gmt");
}

fetch-id.php always checks for a cached copy of the requested gist in the gistscache folder, and if one exists, gets the file’s last modified date and adds it to the headers we’ll pass to the GitHub API.

If the gist has been updated since that date, a status of 200 will be returned along with the latest data. If the gist hasn’t been updated, a 304 will be returned and we’ll serve the cached data.

Next we make the API call and store the status code:

$context = stream_context_create($opts);
$content = file_get_contents("https://api.github.com/gists/$gist_id", false, $context);

$status = getStatus($http_response_header);

Add then we handle the API response:

if ($status === 200) {
  echo $content; // Serve JSON returned from API
  writeFile($cached_file_path, $content); // Add/update cached copy
} else if (
  $status === 304 && $cached_file_exists || // Not modified and we have a cached version
  $status === 403 && $cached_file_exists // We've likely reached our API limit (403 == Forbidden Gist)
) {
  echo file_get_contents($cached_file_path); // Serve cached copy
} else {
  echo "{\"error\": \"Cannot fetch gist. Error code: $status\"}";
}

If we get a 200 — meaning the gist has been updated since the last updated date we sent in the request headers — serve the returned JSON and write a copy to the cache. If we get 304 — meaning the gist hasn’t been updated — serve a cached copy. If we get a 403, it means we’ve reached our API limit, so also serve the cached copy. In any other case, serve an error message in JSON format.

Folder structure

In summary, the gists folder on your proxy server should look something like this:

.
├── .htaccess
├── cached.json
├── fetch-id.php
├── fetch.php
├── gistscache
│   ├── 0276d43b831af40d1bbe529549a66f84.json
│   ├── 2277dd0362789957fd5ce9ed4894c93b.json
│   ├── 35841f68de35bce70ea1bb4cd71ac5d1.json
│   └── [etc.]
└── headers.inc

Building the front end

If you’ve not done so already, grab the code for the front end by cloning get-gists on your local machine:

git clone https://github.com/donbrae/get-gists.git

cd into the get-gists folder and run npm install.

Next, start a local server with npm run start. The page should open in your default browser.

Let’s dig into the code.

Installing a custom build of highlight.js as a npm dependency

We use highlight.js to add syntax highlighting to gists that contain code. By default highlight.js is over 1 MB in size, so to minimise the amount of JavaScript we serve to the user, I created a custom build with only the languages I need, namely JavaScript, HTML, CSS, PHP and Markdown.

You’d normally install highlight.js from the public npm registry via npm install highlight.js, but I instead followed these steps:

1. cd to your development folder and clone the highlight.js repo: git clone https://github.com/highlightjs/highlight.js.git
2. cd highlight.js
3. npm install
4. Run a build for just the languages we need: node tools/build.js javascript xml css php markdown (xml includes HTML)
5. The files we create are in ./build. Run ls -l ./build to list them
6. cd to the root of the project folder you want to use the custom build with (in my case, get-gists)
7. mkdir src to create a src folder
8. mkdir src/highlight.js to create a folder for the highlight.js custom build
9. cd src/highlight.js
10. Use cp to copy over the custom build we created in step 4, eg cp ../../../highlight.js/build/highlight.js ./
11. Run npm init to set the folder up as an npm dependency
12. Answer the prompts. Call it highlight.js and make sure entry point is set to highlight.js (it should be auto-selected)
13. cd ../.. to root of your project folder
14. Run npm install ./src/highlight.js
15. You should now see highlight.js referenced as a dependency in your project’s main package.json: grep --color=always -e "^" -e "highlight.js" package.json
16. highlight.js can now be imported from within a JavaScript file in your project: import hljs from "highlight.js";

index.html

You’ll see in the repo that index.html is a basic HTML page with a button — which, when clicked, will initiate a fetch() of the gist data — and an empty <div> where we’ll add the returned data. We also include the main script, index.js:

<!DOCTYPE html>
<html>
  <head>
    <title>Get gists</title>
    <meta charset="UTF-8" />
  </head>

  <body>
    <h1>Get gists demo</h1>
    <button id="get-gists">Get gists</button>
    <div id="gists" class="hide fade"></div>

    <script src="src/index.js" type="module"></script>
  </body>
</html>

index.js

Note that, here, the two fetch calls we make call the GitHub API directly, but in production these can be swapped for the proxy URLs.

First in index.js we import our CSS files and JavaScript modules:

import './styles.css';
import './xcode.css';
import hljs from 'highlight.js';
import markdownit from 'markdown-it';

• styles.css is some basic CSS for styling the page that lists the gists
• xcode.css is a set of CSS classes that style the code to look like Apple’s Xcode IDE
• highlight.js is our custom build of the code-highlightling module
• markdown-it transforms gists written in Markdown to HTML

Next we define a couple of functions. For security reasons, escapeHtml() sanitises any HTML that is returned by the API, replacing certain characters with their HTML entity equivalents:

// https://stackoverflow.com/a/6234804
function escapeHtml(unsafe) {
  return unsafe
    .replaceAll('&', '&')
    .replaceAll('<', '&lt;')
    .replaceAll('>', '&gt;')
    .replaceAll('"', '&quot;')
    .replaceAll("'", '&#039;');
}

show() handles element fade-ins:

function show(el) {
  el.classList.remove('hide');
  setTimeout(() => {
    el.classList.add('show');
  }, 30);
}

Next, the cfg object defines various properties, such as the GitHub user account we’re fetching gists for, and the IDs of any gists we wish to exclude from our listing:

const cfg = {
  githubUser: 'donbrae',
  hideIds: [
    // IDs of Gists to exclude from page
    '2369abb83a0f3d53fbc3aba963e80f7c', // PDF page numbers
    'bfbda44e3bb5c2883a25acc5a759c8fc', // Bootstrap 5 colour gradient
    'ab4e15be962602b1bf4975b912b14939' // Apple Music shortcuts
  ],
  perPage: 15, // Number of gists to fetch from API
  gistsLimit: 10 // Maximum number of gists to add to page
};

So that we can display the name of the month next to each gist, we define an array of month name abbreviations:

const months = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'];

Next, we have a regular expression to parse text for URLs so we can turn them into links:

const urlRegEx = /(\b(https):\/\/[-A-Z0-9+&@#%?=~_|!:,.;]*[-A-Z0-9+&@#%=~_|])/gi; // Only transform https URLs. Source: https://www.codespeedy.com/replace-url-with-clickable-link-javascript/

We also grab the HTML element into which we’ll place the returned gists:

const gists = document.getElementById('gists');

And then add an event listener to the ‘Get gists’ button:

document.getElementById('get-gists').addEventListener('click', getGists);

This calls our main function, getGists():

function getGists(e) {
  const btnGetGists = e.target;
  btnGetGists.classList.add('fade'); // Fade out button

  function error(err, container, e) {
    console.error(err);
    const escapedHTML = escapeHtml(`${err.status} ${err.statusText}: ${err.url}`);
    container.innerHTML = `<div>${err.status} ${err.statusText}: ${err.url}</div>`;
    show(container);

    const button = e.target;
    button.parentNode.removeChild(button);
  }

  fetch(
    `https://api.github.com/users/${cfg.githubUser}/gists?per_page=${cfg.perPage}`
  )
    .then(function (response) {
      if (response.ok) return response.json();

      return Promise.reject(response);
    })
    .then(function (data) {

      if (data.error) {
        console.error(data.error);
        return;
      }

      /**
       * Pseudocode block 1: show listing of gists returned by API
       */
      // Filter returned items to remove specific gists by ID
      // Loop through filtered gists
      //   Get created date
      //   Get description
      //     Convert any URLs within description to links
      //     Transform backticked text to <code> elements
      //   Create <div> for gist with date, description and 'Get gist' button
      //     Add to button the gist ID as a `data` attribute
      // End loop
      // Add gists to DOM
      // Fade in gists

      /**
       * Pseudocode block 2: show specific gist's content
       */
      // Loop through 'Get gist' buttons in DOM
      //   Add on-click function
      //     Fetch 'https://api.github.com/gists/<gist-id>'
      //       Get gist type
      //         If type is text/html
      //           Escape the HTML
      //         If type is text/markdown
      //           Run the content through markdownit
      //       Add to DOM
      //         If text/markdown
      //           Add gist content in a <div>
      //         Else
      //           Add to UI in a code block
      //           Run code block through highlight.js
      //   End on-click function
      // End loop
      // Fade in gist
    })
    .catch((err) => {
      error(err, gists, e);
    });
}

The fetch() function is the same in structure as the one we tested at the start of this post.

You’ll see a couple of blocks of pseudocode which run when the API call is successful. I find that writing such a high-level overview of the various steps in a process helps me think through a problem without getting bogged down in specifics. Now that we’ve outlined the main steps in pseudocode, we can convert them to real code:

// [...]
      /**
       * Show listing of gists returned by API
       */
      const items = [];

      // Filter returned items to remove specific gists by ID
      const dataFiltered = data.filter(
        (gist) => cfg.hideIds.indexOf(gist.id) === -1
      );

      for (let i = 0; i < cfg.gistsLimit; i++) { // Loop through filtered gists
        if (dataFiltered[i]) {

          // Get created date
          const date = new Date(dataFiltered[i].created_at);
          const dateFormatted = `${date.getDate()} ${
            months[date.getMonth()]
          }`;
          const verb = !i ? 'Created ' : '';
          const year = ` ’${date.getFullYear().toString().slice(-2)}`;

          // Get description
          let description = dataFiltered[i].description.trim().length
            ? `<div>${dataFiltered[i].description}</div>`
            : '';

          // Convert any URLs within description to links
          description = description.replace(urlRegEx, function (match) {
            return `<a href="${match}">${match}</a>`;
          });

          // Transform backticked text to <code> elements
          description = description.replaceAll(/`(.+?)`/gi, function (match) {
            return `<code>${match.slice(1).slice(0, -1)}</code>`;
          });

          // Create <div> for gist with date, description and 'Get gist' button
          items.push(
            `<div class="gist-container">
              <h2><a href="${
                dataFiltered[i].html_url
              }">${
              Object.keys(dataFiltered[i].files)[0]
            }</a></h2> <span class="dt-published">${verb}${dateFormatted} ${year}</span>${description}
              <button class="get-gist display-block button button-sm mt-1" data-gist-id="${
                dataFiltered[i].id // Add to button the gist ID as a `data` attribute
              }">Show</button>
              <div id="gist-${dataFiltered[i].id}" class="gist-content hide fade"></div>
            </div>`
          );
        } else break;
      }

      gists.innerHTML = items.join(''); // Add gists to DOM
      btnGetGists.parentNode.removeChild(btnGetGists); // Remove 'Get gists' button
      show(gists); // Fade in gists

      /**
       * Show specific gist's content
       */

      // Loop through 'Get gist' buttons in DOM
      const getGists = document.querySelectorAll('.get-gist');
      getGists.forEach((getGistButton) => {

        // Add on-click function
        getGistButton.addEventListener('click', (e) => {
          const gistId = e.target.dataset.gistId;
          const btn = e.target;
          btn.disabled = true; // Make sure user only triggers one API call
          btn.classList.add('fade');

          const el = document.getElementById(`gist-${gistId}`);

          // Fetch 'https://api.github.com/gists/<gist-id>'
          fetch(`https://api.github.com/gists/${gistId}`)
            .then(function (response) {
              if (response.ok) return response.json();

              return Promise.reject(response);
            })
            .then(function (data) {

              if (data.error) {
                console.error(data.error);
                el.innerHTML = `<div class="gist-content">${escapeHtml(data.error)}</div>`;
                btn.parentNode.removeChild(btn);
                show(el);
                return;
              }

              const gistName = Object.keys(data.files)[0];
              const gist = data.files[gistName];
              let gistContent;

              // Get gist type
              if (gist.type === 'text/html') { // If type is text/html
                gistContent = escapeHtml(gist.content); // Escape the HTML
              } else if (gist.type === 'text/markdown') { // If type is text/markdown
                const md = new markdownit('default', { html: true }); // Run the content through markdownit
                gistContent = md.render(gist.content);
              } else {
                gistContent = gist.content;
              }

              // Add to DOM
              if (gist.type === 'text/markdown') { // If text/markdown
                el.insertAdjacentHTML( // Add gist content in a <div>
                  'beforeend',
                  `<div>${gistContent}</div>`
                );
              } else {
                el.insertAdjacentHTML( // Add to UI in a code block
                  'beforeend',
                  `<pre class="code" role="code">${gistContent}</pre>`
                );
                hljs.highlightElement(el.querySelector('pre')); // Run code block through highlight.js
              }

              btn.parentNode.removeChild(btn);
              show(el); // Fade in gist
            })
            .catch((err) => {
              error(err, el, e);
            });
        });
      });
// [...]

Summary

And that’s it! To summarise, we:

• used the GitHub API to fetch a listing and contents of public gists for a specific user
• created two proxy server scripts to authenticate with the GitHub API and fetch the data
• added a CORS directive to the proxy server to allow our calling domain to request data
• implemented basic caching to avoid unnecessary API data fetches
• added a couple of npm modules to tranform gists written in Markdown to HTML, and highlight the syntax of gists comprising code. The syntax highlighting module was a custom-built npm dependency to avoid serving the user unnecessary JavaScript

It turns out, however, that the Web Audio API’s AudioContext interface has a new property that returns an estimate in seconds of the output latency. The property, outputLatency, is currently supported in newer versions of Firefox and Chrome¹, and allows us to delay starting the visuals and have them be in sync with the audio.

Testing it out

You can test it by running the code below in the browser console.

First check the latency while your audio output is set to the built-in speakers or a pair of wired headphones:

var audioCtx = new AudioContext()
audioCtx.outputLatency

I get 0 (zero) when I just checked in Firefox on macOS.²

Now set the audio output to a pair of Bluetooth headphones and check the latency again:

audioCtx.outputLatency

I get 0.17780041666666666 seconds.

If I then switch back to the built-in speaker it‘s 0.02485258333333333 seconds. I can’t explain that; feel free to leave a comment if you know the reason.

You can alternatively test it on an updated version of the animated keyboard (below). Output latency will be shown beneath the controls when you press Play.

Safari and older browsers

Because browser support is incomplete, when using outputLatency we should first check that the property exists:³

const hasOutputLatency = window.AudioContext && 'outputLatency' in window.AudioContext.prototype ? true : false;

if (hasOutputLatency) {
  console.log('AudioContext.outputLatency is available');
}

Regarding Safari, I see there is a fixme in the WebKit code, so hopefully the feature will be available soon on iPhones and iPads.

Further reading

There is some great technical background in this post by a Paul Adenot, an engineer at Mozilla who added outputLatency (along with baseLatency and getOutputTimestamp) to Firefox.

Calendar

The first is the cal program, which prints an ASCII calendar.

cal by itself returns the current month with today’s date highlighted.

cal 2022 returns a calendar for the whole of 2022.

cal -3 returns the current and surrounding months. This may be the most useful default view for me:

It also works for the past: cal 1981 returns a calendar for 1981. You can also capture just a month: cal may 1981.

The weather

Another user brought up wttr.in, a service which provides a weather report in ASCII format.

You can run a simple curl wttr.in to retrieve a forecast based on your location:

Assigning alias weather to curl wttr.in

To make the curl wttr.in command more memorable, I decided to assign the alias weather to it. Adding aliases can be useful in general, so I thought I’d document the process:¹

1. cd to your home directory: cd ~/
2. Open the file .bashrc in your preferred text editor (which is nano for me): sudo nano .bashrc
3. Add a new line with an alias: alias weather='curl wttr.in'
4. Save the change (Control-O in nano), then exit (Control+X)
5. Run source .bashrc to add the function to the current terminal session²
6. You can now run weather to get a weather forecast for your current location

Passing an argument for location

wttr.in also lets you query the weather in a specific location. I updated my weather command to allow for this:

1. In your home folder, run unalias weather to remove the alias we just set
2. Open .bashrc in your text editor
3. Delete the line with alias weather='curl wttr.in'
4. Add a weather() function (you don’t need the alias keyword):

weather() {
  if [ $# -eq 0 ] # If no argument has been passed to this function
    then
      curl wttr.in
    else
      curl wttr.in/"$1" # Append location
  fi
}

5. Save and exit
6. Run source .bashrc
7. Now, by adding a location argument to weather you can see what the weather is like in, say, Glasgow (weather glasgow+scotland) or New York (weather new+york).³ weather with no argument returns a forecast based on your location.

There didn’t seem to a solution that matched quite what I was looking for, so I decided to roll my own, a demo of which you interact with below.

Approach

I wanted the solution to be no more complex than it needed to be, so I started with a piano keyboard SVG I found on Wikimedia Commons. I extended the graphic to cover all 88 keys, rounded the key edges, and added some gradients.

My idea was to have the program play an audio file and update the fill property of whichever <rect>s in the SVG represented the keys that were being played.

MIDI to JSON

MIDI was the obvious way to get the music (which I played and recorded in Logic) in the form of note-on, note-off and velocity data. I used Tone.js (‘a Web Audio framework for creating interactive music in the browser’) and Tone.js Midi to covert MIDI file data into a JSON format that could be understood by Tone.js.

When the user clicks ‘Play’, the MIDI events are scheduled by Tone.js; then, when the audio file has loaded, the audio and MIDI files are played simultaneously.

There isn’t a synth hooked up to the Tone.js instance, so the MIDI file is, in effect, silent. Using Tone.js’s Draw class, you can schedule code to run on each MIDI event, and I use this to add and remove note-on class names to the relevant UI elements.

Audio

I use Howler.js, a library built on the Web Audio API, to handle the playing of audio. Here it plays the examples in either M4A or WebM formats, depending on the browser.

UX

The colour red is used to show what the right hand is playing, and blue the left. The brighter the colour, the higher the velocity (i.e. the speed with which the key is being depressed).¹

Additionally, a circle appears above each key as it’s played: a filled-in circle for the right hand, and a circle outline for the left. The circles have a ~1s fade-out to help orient the user. (By orient I mean that when a given note starts playing, there is an indication to the user as to which note or notes were played immediately before.)

The tutorials feature a musical chart (score), often with explanatory annotations, for each example. The site is aimed mostly at players who already have at least some experience of playing piano, and such folk can probably already read Western music notation. To these users the animated keyboards may have limited value.

Even if you can read music, however, seeing the velocities indicated visually might he helpful if you’re looking to improve your command of the jazz or funk ‘feel’. For this funk groove, for example, the colours could help users see which notes are being ‘ghosted’.

Technical details

Playing audio when an iOS device is muted

On iOS, the Web Audio API can’t produce sound if the device is muted. This is probably a feature (rather than a bug) to prevent websites from playing sound when the user doesn’t expect it.

In the case of JazzKeys.fyi, users are explicitly tapping a ‘Play’ button and so very likely want to hear something even if their mute switch is active.

Unlike the Web Audio API, the HTML5 <audio> element plays sound no matter a device’s mute state, and so we can work around the no-sound-on-mute issue by simultaneously playing a silent audio file via an <audio> element.

Code structure

In a typical JazzKeys.fyi tutorial, there will be several musical examples. I structured this in code by creating the class ItemAudioState, from which a new object per example is instantiated on page load:

class ItemAudioState {
  constructor(
    name,
    buttonPlay,
    loop,
    tempo,
    filePath,
    midi,
    numberOfKeys
  ) {
    this.name = name; // e.g. lick-blues-1. Taken from `data-name` on .button-play
    this.buttonPlay = buttonPlay;
    this.loop = loop; // true/false
    this.tempo = tempo; // integer/undefined
    this.filePath = filePath; // e.g. /audio/lick-blues-1/lick-blues-1-straight-120 (file extensions added later)
    this.midi = midi; // Path to MIDI file
    this.numberOfKeys = numberOfKeys; // Number of keys on keyboard
  }
}

State management

Each instantiated instance of ItemAudioState is stored in the state object, in the itemsState property:

const state = {
  init: false,
  itemsState: {}, // Instantiated `ItemAudioState` objects
  activeItemState: undefined, // Reference to active item's object in `itemsState`. Active item == currently playing item (set on Play button click)
  audio: undefined, // Main howler object
  playing: false,
  loading: false,
  playingItemObjectName: undefined, // Name of the item currently being played
  playCount: 0, // Incremented after end of each play of main audio file
  updateAudio: false, // Was a relevant parameter (e.g. tempo or rhythm) changed during playing that requires the main audio file playing to be updated?
  isiOS: undefined, // true/false
  htmlAudio: undefined, // For iOS
  htmlAudioSetup: false, // Flag. Background HTML5 element created
  htmlAudioPlaying: false, // Flag. Background HTML5 has started playing
  tonejs: {
    itemsMidi: {}, // MIDI files as JS objects
    visuals: []
  },
  timeouts: {
    // Store timeouts in state object so they can be cleared (and we can prevent multiple instances thereof being fired). All timeouts in this object are cleared by playStopHowl()
    loading: undefined // If audio taking more than a brief moment to load, show 'loading' indicator
  }
};

Listeners are added to various UI elements, such as the ‘Loop’ switch and ‘Tempo’ dropdown menu. When there is a change in value to one of those UI elements, a matching property in the relevant object in state.itemsState is updated.

A listener is also added to the body to capture any change that‘s made in the UI. This updates the audio file path based on changes to tempo and rhythm (whose values themselves are updated by the element-specific listeners described in the previous paragraph). It also captures changes to elements added dynamically, such as the ‘Bar dimming’ switch².

Querying the state

The state can be updated at any time, including while audio is being played and the keyboard is animating. The updated state will then be queried at the relevant point in the code, such as when the program decides whether the audio should loop and/or load new audio.

The state management also handles the scenario in which the user interrupts the playing of one example by clicking ‘Play’ on another example. Here, the object in state.itemsState which corresponds to the newly activated example will become the source of what audio is to play and what keyboard is to animate.

HTML data attributes

The id and various data attributes in the HTML associate the ‘Play’ button with its accompanying UI elements:

<div class="flex controls">
  <button class="button button-play" data-name="figure-bebop-6" data-tempo="#tempo6" data-chord_name="#chord6" data-loop="#loop6" data-annotations="#annotations6" data-midi="88|49">Play</button>
</div>
<div class="flex controls">
  <div>
    <label for="tempo6">Tempo (speed)</label>
    <select id="tempo6" class="select tempo" data-name="figure-bebop-6">
      <option value="140">140bpm (1x)</option>
      <option value="105">105bpm (0.75x)</option>
      <option value="70">70bpm (0.5x)</option>
    </select>
  </div>
  <div>
    <label for="chord6">Chords</label>
    <select id="chord6" class="select chord-name" data-name="figure-bebop-6">
      <option value="">None</option>
      <option value="C7|F-rh-140">C7 | Fmaj7</option>
      <option value="Gm7-C7|F-rh-140">Gm7 C7 | Fmaj7</option>
    </select>
  </div>
  <div>
    <label class="form-switch">
      Loop
      <input type="checkbox" class="loop" id="loop6" data-name="figure-bebop-6">
      <i></i>
    </label>
  </div>
  <div>
    <label class="form-switch">
      Annotations
      <input type="checkbox" class="annotations" id="annotations6" checked data-name="figure-bebop-6">
      <i></i>
    </label>
  </div>
</div>

That all elements for each example share the same name data attribute allows the JavaScript to target the relevant object in state.³

Get notified on launch

And there we have it: a simple animated piano keyboard built in JavaScript. You can view the demo and edit its code at this CodeSandbox.

If you’d like to be notified when JazzKeys.fyi launches, feel free to enter your email.

macOS native shortcuts and modifiers

Useful shortcuts already defined by the system.

Window management

• Command-Backtick: cycle current application’s windows
• Command-H: hide active application’s windows¹
• Command-Option-H: hide all windows expect those belonging to the active application’s^1:1
• Hold Option while resizing window: change window dimension(s)² around the window’s current centre point
• Hold Option-Shift while resizing window: change window dimensions around the window’s current centre point while maintaining aspect ratio

(See also Rectangle Pro.)

Special characters

• Hold $KEY in text fields to get pop-up menu of given character with diacritic markings
• Control-Command-Space bar: open Character Viewer (emoji and symbols picker)
• Option-Hyphen: insert en dash (–)
• Option-Shift-Hyphen: em dash (—)
• Option-Close square bracket: opening single curly quote (‘)
• Option-Shift-Close square bracket: closing single curly quote (’)
• Option-Open square bracket: opening double curly quote (“)
• Option-Shift-Open square bracket: closing single curly quote (”)

You can also search the Character Viewer (Control-Command-Space bar) by name, for ‘en dash’ etc.

‘Hidden’ functionality

Holding the Option key while clicking or hovering over a UI element can reveal additional functionality or information. Some examples: the Wi-Fi and Bluetooth menubar icons; the ‘Plus’ button when building a Finder search (allowing you to enhance the search logic); and application dropdown menus in the menu bar.

One I find useful is ‘Copy as Pathname’ in the Finder Edit menu:

Custom OS mappings

You can assign application keyboard shortcuts in System Preferences > Keyboard > Shortcuts > App Shortcuts:

• Control-Option-Command-M in Finder: run Window > Merge All Windows
• Command-Option-I in Preview: Adjust Size… from the Tools menu
• Shift-Command-Forward slash in all apps: ‘Show Help menu’ (to search menu items):

Rectangle Pro

Third-party window manager. Includes window snapping à la Microsoft Windows. The keyboard shortcuts I use:

• Shift-Command-Enter: current window to medium size, centred (custom shortcut)
• Command-Option-Enter: ‘Almost Maximize’
• Control-Option-Command-Enter: full screen, with some padding, centred (custom shortcut)
• Control-Option-Command-Left Arrow: ‘Left Half’
• Control-Option-Command-Right Arrow: ‘Right Half’
• Control-Option-Command-Down Arrow: ‘Center’
• Control-Option-Command-Escape: ‘Restore’

Karabiner-Elements

Third-party keyboard customiser that I use to open and switch between commonly used apps with a single keystroke. It’s generally more efficient than using a trackpad or mouse, Spotlight, LaunchBar or Command-Tabing.

I generally assign Control-Shift-$KEY to these.

• Control-Shift-U: launch/switch to Apple Music
• Control-Shift-C: Calendar
• Caps Lock³: Finder
• Control-Option-Shift-D: Downloads folder in Finder
• Control-Shift-F: Firefox
• Control-Shift-D: Firefox Developer Edition
• Control-Shift-G: Google Chrome
• Control-Shift-I: iTerm
• Shift-Caps Lock: Logic Pro
• Control-Shift-M: Microsoft Teams
• Control-Shift-N: Nova
• Control-Shift-S: Safari
• Control-Shift-P: Spotify
• Control-Shift-T: Terminal
• Control-Shift-R: Transmit
• Control-Option-Shift-V: Visual Studio
• Control-Shift-V: Visual Studio Code

Assign shortcuts by editing profiles[0].simple_modifications in ~/.config/karabiner/karabiner.json⁴. For example, to open Firefox Developer Edition:

{
	"from": {
		"key_code": "d",
		"modifiers": {
			"mandatory": [
				"left_shift",
				"left_control"
			]
		}
	},
	"to": [
		{
			"shell_command": "open /Applications/Firefox\\ Developer\\ Edition.app"
		}
	]
}

Or to open Finder (or whatever you have assigned as your default file browser):

{
	"from": {
		"key_code": "caps_lock"
	},
	"to": [
		{
			"consumer_key_code": "al_local_machine_browser"
		}
	]
}

Open several Finder windows at specific locations

I use Control-Option-Shift-J to open several Finder windows⁵ at locations relevant to my JazzKeys.fyi project:

{
	"from": {
		"key_code": "j",
		"modifiers": {
			"mandatory": [
				"left_control",
				"left_shift",
				"left_option"
			]
		}
	},
	"to": [
		{
			"shell_command": "open ~/Documents/dev/JazzKeys.fyi\\ process\\ files\\ \\(Hazel\\); open ~/Documents/dev/jazztoolkit/site/audio; open ~/Library/Mobile\\ Documents/com~apple~CloudDocs/Logic\\ projects/jazzkeys.fyi/Jamming; open ~/Library/Mobile\\ Documents/com~apple~CloudDocs/Logic\\ projects/jazzkeys.fyi/Licks\\ etc."
		}
	]
}

Third-party app-specific

Some third-party apps which I find useful enough to assign keyboard shortcuts to. The shortcuts are defined within the apps themselves.

I generally assign Control-Option(-Command)-$KEY to these.

• Control-Option-C: open Copy ’Em (clipboard manager)
• Control-Option-F: Dato⁶ (menubar calendar app)
• Control-Option-Command-Space: DevUtils.app (developer tools)
• Control-Option-Command-F: Dropzone (file management and general productivity booster)
• Control-Z: Menuwhere (get the Mac menu bar wherever your cursor is)
• Control-Option-Command-N: Quick Note > new note
• Shift-Command-2: Snappy (take screenshots which remain in front of other windows) > new snap
• Control-Option-Command-C: System Color Picker (OS-wide access to an enhanced system colour picker) > pick colour

(I also use aText for text automation/expansion, but that’s probably a separate blog post.)

I wrote the original prototype in 2013 using jQuery, but it’s now 2021 so the first task was to refactor the codebase to use vanilla JavaScript.² For the animations — previously handled by jQuery’s animate() function — I ended up using the Web Animations API. It’s still at Working Draft status, but widely supported. I actually began by using the top/left position properties to move elements, and specifying the animation transitions in the CSS, but the frame rate across devices was patchy. The switch to using the WAAPI and translate() CSS function — which leverages hardware acceleration — resulted in improved performance and more fine-grained control of animations.

Here’s a basic example of the WAAPI and translate() in action³:

See the Pen Web Animations API by donbrae (@donbrae) on CodePen.

The howler.js library is used for audio⁴, and canvas-confetti for the ‘win’ animation.

The game works with both pointer and touch input, and can more or less function as a Progressive Web App (PWA) on mobile devices by way of iOS-specific metatags, favicons, splash screen graphics (iOS only), and a manifest.json file. (Offline access still needs to be implemented.) When added to the home screen on my iPhone it pretty much behaves as if it was a native app.

I built it mainly using VS Code and Firefox Developer Edition, with GitHub for version control and Netlify for hosting and easy git-based deployment. (Oh, and I created the sounds in Logic.)

Overall, this was a challenging and enjoyable side project. I started my programming journey writing games in Sinclair BASIC, so the process has had nostalgia value alongside giving me a chance to hone my dev chops. For anyone who’s interested, the code is available to view on GitHub, and you can play the game here.

If memory serves, what used to be sufficient was adding user-scalable=no and maximum-scale=1 to the ‘viewport’ meta tag:

<meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1, maximum-scale=1">

However this certainly doesn’t work on iOS now, and using it will lower your Google Lighthouse score.

It is nonetheless still possible to disable zooming.

touch-action: none; prevents pinching to zoom (source: MDN):

#game_surface {
	touch-action: none;
}

(I’ve actually applied this rule to html and body, rather than just the game surface.)

And you can disable double-tap-to-zoom with JavaScript (source: Stack Overflow):

let last_touch_end = 0;
document.addEventListener("touchend", function (e) {
  const now = (new Date()).getTime();
  if (now - last_touch_end <= 300) {
      e.preventDefault();
  }
  last_touch_end = now;
}, false);

You might also wish to disable text selection:

#game_surface {
	touch-action: none;
    -webkit-user-select: none;
    user-select: none;
}

It’s worth repeating that you shouldn’t add these restrictions to web pages and apps by default, as it compromises accessibility. But for a game or similar application where you do want to prevent default zooming, the above code does the job.

I remember earlier on in my musical journey thinking it would be great if I could just find a ready collection of blues licks to quickly ‘level up’ my playing (my command of blues was seriously lacking at that point). Same thing when I later found myself in a funk band and knowing not much about playing funk keyboards. I also long struggled to sound ‘modern’ and find my own contemporary jazz voice.

This new site aims to be a resource for other players out there who may be struggling in some way or who just want to augment their playing. As per the landing page, there will be written scores for those that can read Western music notation; and audio too, with the option to slow down and loop the audio, as well as listen with chordal accompaniment.

Like this blog, it’s a Jamstack site built with the Supermaya Eleventy starter kit, and is hosted on Netlify. The howler.js library is used to make working with the Web Audio API less of a fash.

Check it out, and add your email if you’d like to be notified when it launches proper.

Update: I’ve published a tutorial on bebop enclosures.

A workaround is to use Web Audio but simultaneously play a silent audio file via HTML5 Audio.

This can be the subject of a future post. But first I needed to work out whether the user is running iOS. It’s simple enough, complicated only slightly by the fact that iPads run a version of desktop Safari. Here’s the code I’m using:

const user_agent = navigator.userAgent.toLowerCase();
const is_iOS =
    user_agent.indexOf("iphone") > -1 ||
    user_agent.indexOf("ipod") > -1 ||
    user_agent.indexOf("ipad") > -1 || // This may not be required
    (navigator.maxTouchPoints && /Mac/.test(navigator.platform)); // iPad running 'desktop' Safari

is_iOS returns truthy or falsey, letting us know whether the user is on iOS.

Good practice for tailoring your code to a particular browser is to use feature detection (as we do here with navigator.maxTouchPoints). However additionally querying navigator.userAgent for the presence of a string is fine too when we can’t rely soley on feature detection. Of course you may need to update the query in future if user agent or platform names change.

(An alternative to the playing-silent-HTML5-Audio workaround would be to instead read is_iOS’s value and print a helpful ‘Unmute your device’ message under the ‘Play’ button. Unfortunately there’s no API call you can make to get the mute state.)

You can run the iOS detection code here (courtesy of CodeSandbox).

The data is currently in the form of a JavaScript object literal, and I wanted to check whether feeding Fuse JSON instead would result in faster searches (as I jaloused it might).

Fuse.js is already plenty fast but I wanted to identify further opportunites for optimisation as I (slowly) add more words to the dictionary. (And why not make the app as efficient as possible anyway?)

So I ran a few simple tests on three desktop browsers. Each test does 300 searches (via the ‘choose a random word’ button) per browser per JS object literal and JSON file, the latter of which is 134 kB. The maximum number of possible values to check against in the dataset is 4,239.

The results, averaged across each of the six tests, are:

So I was wrong in my hypothesis: searching the JSON is in fact slower, by as much as 10% in the case of Safari.

As far as comparing the browsers, in these tests the Chrome V8 JavaScript engine performs about two-and-a-half times as fast as Safari and Firefox’s.

In terms of actually parsing larger amounts of data, this article pegs JSON as the winner, but the above tests show that for my use case — using Fuse.js to fuzzy-search a dataset with about 4,000 possible items to check against — object literals appear to be the way to go.

(There’s a CodeSandbox here with the actual results data.)

To provide a real-time experience the app checks the Google API and RSS output asynchronously at regular intervals: the user is notified in a button that there are ‘x new updates’. It aims to be similar to the Guardian’s ‘Live’ pages.