First mockup: Gen tree @ the Met

Sarah Schwettmann; Chris Hoder

First storyboard:

1. Evo algorithm guides visitor through a 3x3 grid of images selected from the Met collection. User selects the most [surprising]. Based on selection, user sees 9 more images, perhaps including gen works. Process repeats [5] times.

2. Trained model generates artwork similar to the selected 5. Visitor sees gen artwork + the 5 images in its family tree. At this point, visitor can {print gen artwork, explore full tree* of gen work, enter gen context}

*Tree: artwork graph with family tree of gen works. Starting nodes are Met artworks, paths show their breeding. Based on this graph, visitors can be shown works that are similar to their gen work in the *existing* Met collection.

Pseudocode

EXPLORING

Image-grid-selector (image-array): [return image-array]
Choose 9 images from the Met collection,
non-repeating (option: with filtering)
randomly chosen from? chosen to be maximally different from?
the existing image-array of past chosen images.

The visitor selects one image from the grid,
adds it to image-array,
and runs image-grid-selector again
until you have 5 images in the image-array
Show the already-selected images, smaller, above.

Gen-image-editor (image-array)

Display-image-grid (image-array)
GAN-surfer (image-array) [or GAN-parameter-tweaker]
Display-image-details (GAN-id)

Display-image-grid (image-array)
For i in image-array: show-tags(i); show-thumbnail(i)

Display-image-details (GAN-id)
Show title, details from GAN-id metadata
Show form for updating GAN-id metadata
Show buttons: [print poster], [catalog], [story]

GAN-surfer(image-array)
Show a GAN image based on some function of (image-array)
Offer tools for browsing latent space around the shown image:
By toggling parameters, or by surfing around noise-seeds

Gen-tree-catalog (image-id [gen or Met])
Show a gen image
linked to their 5 ancestor Met images
When clicking on a Met image
show its children, if any
To the side, show thumbnails of the last few gen images
Start w/ image-id
(Optional: also show the top N most-used ancestor images
in a similarity-clustered map)

Flask APIs

Setup

Code Repo: https://chrishoder.visualstudio.com/ChrisHoder/_git/methack
- use the same credentials as the azure subscription
Walkthrough on how to setup, update and run: https://docs.microsoft.com/en-us/azure/app-service/containers/quickstart-python

Test

Created a jupyter notebook for test calling the APIs. Located in the repo here: https://chrishoder.visualstudio.com/ChrisHoder/_git/methack?path=%2Ftests%2FTest%20APIs.ipynb&version=GBmaster

APIs

grid-selector

POST call: http://genartstudio.azurewebsites.net/grid-selector
Expects an input with the following format:

Expects an output with the following format:
JSON returned to UI for 9 images:
{ "images": [
{ "id": "some id 1", "url": "someurl_1", “tags”:[‘a’,’b’]},
{ "id": "some id 2", "url": "someurl_2", “tags”:[‘a’,’b’]},
]
}

generate-image

POST call: http://genartstudio.azurewebsites.net/grid-selector
Expects an input with the following format:

{

"selected":[ { "id": "some id 1", "url": "someurl_1"}],

"category":""

}

Expects an output with the following format:
JSON returned to UI for 9 images:
{ "generated-image":{"url":""},
"selected-images":[{ "id": "some id 1", "url": "someurl_1"}]
}