Subscribe: HPCC Systems
http://hpccsystems.com/bb/feed.php
Added By: Feedage Forager Feedage Grade A rated
Language:
Tags:
code  dataset  dec ecl  dec  ecl •  ecl  error  fri dec  posted  rec  row  statistics posted  tue dec  — fri     
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: HPCC Systems

HPCC Systems



A short text to describe your forum



Updated: 2017-12-15T20:08:42+00:00

 



ECL • Ignore transform on LEFT OUTER join non match.

2017-12-15T20:08:42+00:00

So I was wondering if there's a way to have a LEFT OUTER join not go through the transform function on non-matches.

Currently the following code will blank out records 2 and 3:

Code:
rec := {string a, string b, string c};
DS1 := DATASET([{'1','',''}, {'2','Brian','Ge'}, {'3','Kate','Wins'}, {'4','Greg',''}], rec);
DS2 := DATASET([{'1','Fred','Flintstone'}, {'4','Jerry','Fring'}], rec);

result := JOIN(DS1, DS2,
   LEFT.a = RIGHT.a, TRANSFORM(rec,
   SELF := RIGHT
   ), LEFT OUTER);
OUTPUT(result);

This makes sense because every record goes through the transform and those with no matches are getting set to an empty record.

I can accomplish what I need like so:
Code:
rec := {string a, string b, string c};
DS1 := DATASET([{'1','',''}, {'2','Brian','Ge'}, {'3','Kate','Wins'}, {'4','Greg',''}], rec);
DS2 := DATASET([{'1','Fred','Flintstone'}, {'4','Jerry','Fring'}], rec);

result := JOIN(DS1, DS2,
   LEFT.a = RIGHT.a, TRANSFORM(rec,
   SELF.b := IF(RIGHT.a <> '', RIGHT.b, LEFT.b),
   SELF.c := IF(RIGHT.a <> '', RIGHT.c, LEFT.c),
   SELF := LEFT
   ), LEFT OUTER);
OUTPUT(result);


However this method would require including every single field we want from the RIGHT record. I feel like there might be a better way to do this.

EDIT - I figured out a way to do it nicely:
Code:
rec := {string a, string b, string c};
DS1 := DATASET([{'1','',''}, {'2','Brian','Ge'}, {'3','Kate','Wins'}, {'4','Greg',''}], rec);
DS2 := DATASET([{'1','Fred','Flintstone'}, {'4','Jerry','Fring'}], rec);

result := JOIN(DS1, DS2,
   LEFT.a = RIGHT.a, TRANSFORM(rec,
   SELF := IF(RIGHT.a <> '', RIGHT, LEFT)
   ), LEFT OUTER);
OUTPUT(result);

Statistics: Posted by BGehalo — Fri Dec 15, 2017 8:08 pm





ECL • Java plugin output location

2017-12-15T12:26:14+00:00

If I was writing standard out within the Java that the ECL Java plugin uses where would this output reside in the ECL client once run? I assumed the logs but can't find anything.

Statistics: Posted by wjblack — Fri Dec 15, 2017 12:26 pm





Installation • Re: mythor failed to start

2017-12-12T18:40:47+00:00

It would help if I knew what the error message is telling me:
Code:
DFS cluster topology for 'mythor', does not match existing DFS group layout for group


What I think it is telling me is that there is something in the Topology attribute of my environment.xml file that doesn't match with something about 'mythor' that is stored someplace on /var/lib/HPCCSystems.

Statistics: Posted by tlhumphrey2 — Tue Dec 12, 2017 6:40 pm





Installation • Re: mythor failed to start

2017-12-12T18:36:03+00:00

My last post said I had fixed the problem. But, the fix was removing contents of /var/lib/HPCCSystem. But, I really don't want to do that.

I get this error after starting the cluster's instances on AWS. Everytime. I can get rid of the problem by doing:
Code:
sudo rm -r /var/lib/HPCCSystems/*
which deletes the contents of /var/lib/HPCCSystems.

But, that isn't the best solution because I need data that is stored there.

Anyone have any ideas???

Statistics: Posted by tlhumphrey2 — Tue Dec 12, 2017 6:36 pm





ECL • Re: Dataset too large to output to workunit Exception

2017-12-12T15:30:24+00:00

First, the first parameter is a string, and needs to be quoted.
Second, the number is the number of megabytes to use.
Third, the strings that you give are case sensitive.
Fourth, there are two settings: outputLimit and outputLimitMb. These are synonyms. Both take the size in megabytes.

Working and tested example:
Code:
IMPORT std;
#OPTION('outputLimitMb', 100);
Std.File.LogicalFileList('*', 1, 1, FALSE);


Let's just say we have a lot of files (image)

Statistics: Posted by kovacsbv — Tue Dec 12, 2017 3:30 pm





Installation • Mulit-Disk setup

2017-12-12T15:26:30+00:00

Hi,

Let's say I have 1 physical node with 2 thor slaves. I believe in the typical setup, each slave has it's own memory allocation but write (or spill) to the same disk. Is it possible to have each slave write to a separate disk? Of course this is assuming that my physical node has more than one mounted volumes.

Mike

Statistics: Posted by micevepay — Tue Dec 12, 2017 3:26 pm





ECL • Re: Returning png data from embedded python

2017-12-06T13:48:33+00:00

Hi Tim,

I do not work with Python, but I found some interesting links:

https://stackoverflow.com/questions/7380460/byte-array-in-python

https://www.w3resource.com/python/python-bytes.php

I guess the compiler doesn't like DATA as the return value, maybe try a SET OF UNSIGNED1 ?

Regards,

Bob

Statistics: Posted by bforeman — Wed Dec 06, 2017 1:48 pm





ECL • Returning png data from embedded python

2017-12-05T14:24:56+00:00

I'm trying to return an image from my embedded python. Here is my code:
Code:
IMPORT Python;
DATA ReadAndOutputPngFile() := EMBED(Python)
    from imageio import imread
    import numpy as np

    output_index=65
    png=imread('class%s-maximal-image.png' % (output_index))
    return png
ENDEMBED;

OUTPUT(ReadAndOutputPngFile());


When I attempt to execute this, I get the following error message:
Code:
Error: Error: 0: pyembed: type mismatch - bytearray expected


What must my return data type?

Statistics: Posted by tlhumphrey2 — Tue Dec 05, 2017 2:24 pm





ECL • Re: System error: 28

2017-12-05T11:51:52+00:00

I am seeing this error occasionally, too. If this error code 28 is being returned by the operating system during a write call, it means "no space left on device".

Statistics: Posted by RoqScheer — Tue Dec 05, 2017 11:51 am





ECL • Re: Use of ROW

2017-12-01T17:31:37+00:00

Thanks, Bob.

Statistics: Posted by janet.anderson — Fri Dec 01, 2017 5:31 pm





ECL • Re: Use of ROW

2017-12-01T17:29:15+00:00

Janet,

Any definition that passes parameters is an ECL function, and what is "return"ed is the result of the expression, but there is also a FUNCTION structure that allows you to encapsulate code and RETURN a specific value.

Regarding the ROW usage, since it is inline with a recordset filter, the compiler knows in that case to read a single record. When you pull it out of the filter, ROW no longer knows which specific record you are referring to, and hence the error.

Bob

Statistics: Posted by bforeman — Fri Dec 01, 2017 5:29 pm





ECL • Re: Use of ROW

2017-12-01T16:56:49+00:00

Why does an function like CanSearch not require a RETURN statement?

Since the full statement
Code:
%HoldL_CNPNAME_zipcan% := %ToProcess%(~BizLinkFull.Key_BizHead_L_CNPNAME_ZIP.CanSearch(ROW(%ToProcess%)));

does not give an error, and here ROW is also taking the dataset %ToProcess%, then why would it give me an error when I want to break it down and look at only
Code:
%HoldL_CNPNAME_ziprow% := ROW(%ToProcess%);


To see an example, look at Dataland WU W20171201-114948: lines 367-372 are commented out and the code runs. W20171201-114953 the job fails when the line when I uncomment these lines.

Statistics: Posted by janet.anderson — Fri Dec 01, 2017 4:56 pm





ECL • Re: System error: 28

2017-12-01T16:08:56+00:00

Hi Katy,

Did you figure this out yet? Could it be that you did not have rights to access that cluster?

Regards,

Bob

Statistics: Posted by bforeman — Fri Dec 01, 2017 4:08 pm





ECL • Re: Java Plugin

2017-12-01T16:03:38+00:00

Hi William,

If the STRING usage works, why not return the STRING and then cast it to a BOOLEAN definition after?

IsValidationDOC := (BOOLEAN)ValidationDoc();

Something like that....

Regards,

Bob

Statistics: Posted by bforeman — Fri Dec 01, 2017 4:03 pm





ECL • Re: Use of ROW

2017-12-01T15:54:25+00:00


What is causing the error?
There are 3 forms listed in the language reference for ROW, and this doesn't appear to be following any of them. How is it even working before I try to break down the statement?
The attribute CanSearch takes a parameter, but it's not a function or a macro. What is it?


CanSearch looks like a BOOLEAN function to me, based on the expression.

ROW is looking for a single record input, and it looks like you are providing a whole recordset. Hence the message DATASET[1], which would return a single record and satisfy the ROW requirement.

Regards,

Bob

Statistics: Posted by bforeman — Fri Dec 01, 2017 3:54 pm