close
The Wayback Machine - https://web.archive.org/web/20210115212522/https://resources.wolframcloud.com/NeuralNetRepository/resources/Ademxapp-Model-A1-Trained-on-Cityscapes-Data

Wolfram Research

Ademxapp Model A1 Trained on Cityscapes Data

Segment an image of a driving scenario into semantic component classes

Released in 2016 by the University of Adelaide, this model exploits recent progress in the understanding of residual architectures. With only 17 residual units, it is able to outperform previous, much deeper architectures.

Number of layers: 141 | Parameter count: 124,081,107 | Trained size: 497 MB |

Training Set Information

Performance

Examples

Resource retrieval

Get the pre-trained net:

In[1]:=
NetModel["Ademxapp Model A1 Trained on Cityscapes Data"]
Out[1]=BERJAYA

Evaluation function

Write an evaluation function to handle net reshaping and resampling of input and output:

In[2]:=
netevaluate[img_, device_: "CPU"] := Block[
  {net, resized, encData, dec, mean, var, prob},
  net = NetModel["Ademxapp Model A1 Trained on Cityscapes Data"];
  resized = ImageResize[img, {1024}];
  encData = Normal@NetExtract[net, "Input"];
  dec = NetExtract[net, "Output"];
  {mean, var} = Lookup[encData, {"MeanImage", "VarianceImage"}];
  prob = NetReplacePart[net,
     {"Input" -> 
       NetEncoder[{"Image", ImageDimensions@resized, 
         "MeanImage" -> mean, "VarianceImage" -> var}], 
      "Output" -> Automatic}
     ][resized, TargetDevice -> device];
  prob = ArrayResample[prob, 
    Append[Reverse@ImageDimensions@img, 19]];
  dec[prob]
  ]

Label list

Define the label list for this model. Integers in the model’s output correspond to elements in the label list:

In[3]:=
labels = {"road", "sidewalk", "building", "wall", "fence", "pole", 
   "traffic light", "traffic sign", "vegetation", "terrain", "sky", 
   "person", "rider", "car", "truck", "bus", "train", "motorcycle", 
   "bicycle"};

Basic usage

Obtain a segmentation mask for a given image:

In[4]:=
CloudGet["https://www.wolframcloud.com/objects/1aa691ed-5e72-4ce0-bae7-9332af2117d5"] (* Evaluate this cell to copy the example input from a cloud object *)

Inspect which classes are detected:

In[5]:=
detected = DeleteDuplicates@Flatten@mask
Out[5]=BERJAYA
In[6]:=
labels[[detected]]
Out[6]=BERJAYA

Visualize the mask:

In[7]:=
Colorize[mask]
Out[7]=BERJAYA

Advanced visualization

Associate classes to colors using the standard Cityscapes palette:

In[8]:=
colors = Apply[
  RGBColor, {{128, 64, 128}, {244, 35, 232}, {70, 70, 70}, {102, 102, 
     156}, {190, 153, 153}, {153, 153, 153}, {250, 170, 30}, {220, 
     220, 0}, {107, 142, 35}, {152, 251, 152}, {70, 130, 180}, {220, 
     20, 60}, {255, 0, 0}, {0, 0, 142}, {0, 0, 70}, {0, 60, 100}, {0, 
     80, 100}, {0, 0, 230}, {119, 11, 32}}/255., {1}]
Out[8]=BERJAYA
In[9]:=
indexToColor = Thread[Range[19] -> colors];

Write a function to overlap the image and the mask with a legend:

In[10]:=
result[img_, device_: "CPU"] := Block[
  {mask, classes, maskPlot, composition},
  mask = netevaluate[img, device];
  classes = DeleteDuplicates[Flatten@mask];
  maskPlot = Colorize[mask, ColorRules -> indexToColor];
  composition = ImageCompose[img, {maskPlot, 0.5}];
  Legended[
   Row[Image[#, ImageSize -> Large] & /@ {maskPlot, composition}], 
   SwatchLegend[indexToColor[[classes, 2]], labels[[classes]]]]
  ]

Inspect the results:

In[11]:=
CloudGet["https://www.wolframcloud.com/objects/4ab6288d-2b9d-4845-863b-6899933cf077"] (* Evaluate this cell to copy the example input from a cloud object *)
Out[11]=BERJAYA
In[12]:=
CloudGet["https://www.wolframcloud.com/objects/e530e559-e5e3-4f00-a12b-958773276da7"] (* Evaluate this cell to copy the example input from a cloud object *)
Out[12]=BERJAYA
In[13]:=
CloudGet["https://www.wolframcloud.com/objects/17a1fafd-664b-4913-a81d-b221c8ee5cad"] (* Evaluate this cell to copy the example input from a cloud object *)
Out[13]=BERJAYA

Net information

Inspect the number of parameters of all arrays in the net:

In[14]:=
NetInformation[
 NetModel["Ademxapp Model A1 Trained on Cityscapes Data"], \
"ArraysElementCounts"]
Out[14]=BERJAYA

Obtain the total number of parameters:

In[15]:=
NetInformation[
 NetModel["Ademxapp Model A1 Trained on Cityscapes Data"], \
"ArraysTotalElementCount"]
Out[15]=BERJAYA

Obtain the layer type counts:

In[16]:=
NetInformation[
 NetModel["Ademxapp Model A1 Trained on Cityscapes Data"], \
"LayerTypeCounts"]
Out[16]=BERJAYA

Display the summary graphic:

In[17]:=
NetInformation[
 NetModel["Ademxapp Model A1 Trained on Cityscapes Data"], \
"SummaryGraphic"]
Out[17]=BERJAYA

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[18]:=
jsonPath = 
 Export[FileNameJoin[{$TemporaryDirectory, "net.json"}], 
  NetModel["Ademxapp Model A1 Trained on Cityscapes Data"], "MXNet"]
Out[18]=BERJAYA

Export also creates a net.params file containing parameters:

In[19]:=
paramPath = FileNameJoin[{DirectoryName[jsonPath], "net.params"}]
Out[19]=BERJAYA

Get the size of the parameter file:

In[20]:=
FileByteCount[paramPath]
Out[20]=BERJAYA

The size is similar to the byte count of the resource object:

In[21]:=
ResourceObject[
  "Ademxapp Model A1 Trained on Cityscapes Data"]["ByteCount"]
Out[21]=BERJAYA

Requirements

Wolfram Language 11.3 (March 2018) or above

Resource History

Reference