Skip to content

Release 3.0.0 #156

Closed
Closed

Description

@nirgilboa

Hi !

First off thank you for your work on project, it has been very useful :)
Have there been any issues reported on version 3.0.0-beta0 ?

Thanks,
Nir

Activity

  1. nsano-rururu commented on May 30, 2020

    @nsano-rururu

    With the docker image published on dockerhub, there seems to be no problem.
    https://hub.docker.com/r/bitsensor/elastalert/

    When I try to build a new Docker, I cannot create the Docker image properly with an error.
    The content of Update to python3 and elastalert v0.2.4 # 174 in the pull request must be reflected in the Dockerfile.

    Part of the message

    Sending build context to Docker daemon  150.5kB
    Step 1/29 : FROM alpine:latest as py-ea
     ---> a24bb4013296
    Step 2/29 : ARG ELASTALERT_VERSION=v0.2.0b2
     ---> Using cache
     ---> 80c4512cc9b8
    Step 3/29 : ENV ELASTALERT_VERSION=${ELASTALERT_VERSION}
     ---> Using cache
     ---> 635593853902
    Step 4/29 : ARG ELASTALERT_URL=https://github.com/Yelp/elastalert/archive/$ELASTALERT_VERSION.zip
     ---> Using cache
     ---> 96eafc0ff511
    Step 5/29 : ENV ELASTALERT_URL=${ELASTALERT_URL}
     ---> Using cache
     ---> 714703369180
    Step 6/29 : ENV ELASTALERT_HOME /opt/elastalert
     ---> Using cache
     ---> a44f4cbd57a9
    Step 7/29 : WORKDIR /opt
     ---> Using cache
     ---> 9b6ad7e8eb08
    Step 8/29 : RUN apk add --update --no-cache ca-certificates openssl-dev openssl python2-dev python2 py2-pip py2-yaml libffi-dev gcc musl-dev wget &&     wget -O elastalert.zip "${ELASTALERT_URL}" &&     unzip elastalert.zip &&     rm elastalert.zip &&     mv e* "${ELASTALERT_HOME}"
     ---> Running in 650fa41bdab3
    fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
    fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
    ERROR: unsatisfiable constraints:
      py2-pip (missing):
        required by: world[py2-pip]
      py2-yaml (missing):
        required by: world[py2-yaml]
    The command '/bin/sh -c apk add --update --no-cache ca-certificates openssl-dev openssl python2-dev python2 py2-pip py2-yaml libffi-dev gcc musl-dev wget &&     wget -O elastalert.zip "${ELASTALERT_URL}" &&     unzip elastalert.zip &&     rm elastalert.zip &&     mv e* "${ELASTALERT_HOME}"' returned a non-zero code: 2
    

    When I modify the Dockerfile with the pull requested content and build, I get some warnings.

    Part of the message

    npm WARN deprecated babel-preset-es2015@6.24.1: 🙌  Thanks for using Babel: we recommend using babel-preset-env now: please read https://babeljs.io/env to update!
    npm WARN deprecated istanbul@0.4.5: This module is no longer maintained, try this instead:
    npm WARN deprecated   npm i nyc
    npm WARN deprecated Visit https://istanbul.js.org/integrations for other alternatives.
    npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
    npm WARN deprecated joi@13.7.0: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
    npm WARN deprecated json3@3.3.2: Please use the native JSON object instead of JSON 3
    npm WARN deprecated mkdirp@0.5.1: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
    npm WARN deprecated chokidar@1.7.0: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
    npm WARN deprecated core-js@2.6.11: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
    npm WARN deprecated topo@3.0.3: This module has moved and is now available at @hapi/topo. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
    npm WARN deprecated hoek@5.0.4: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
    npm WARN deprecated fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
    npm WARN deprecated hoek@6.1.3: This module has moved and is now available at @hapi/hoek. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
    npm WARN deprecated circular-json@0.3.3: CircularJSON is in maintenance only, flatted is its successor.
    npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
    npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
    
    > dtrace-provider@0.8.8 install /opt/elastalert-server/node_modules/dtrace-provider
    > node-gyp rebuild || node suppress-error.js
    
    make: Entering directory '/opt/elastalert-server/node_modules/dtrace-provider/build'
      TOUCH Release/obj.target/DTraceProviderStub.stamp
    make: Leaving directory '/opt/elastalert-server/node_modules/dtrace-provider/build'
    
    > core-js@2.6.11 postinstall /opt/elastalert-server/node_modules/core-js
    > node -e "try{require('./postinstall')}catch(e){}"
    
    npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/chokidar/node_modules/fsevents):
    npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
    npm WARN ajv-keywords@2.1.1 requires a peer of ajv@^5.0.0 but none is installed. You must install peer dependencies yourself.
    

    In order to reduce warnings, it has been confirmed that Elasticsearch 7.7.0 / Kibana 7.7.0 will alert slack by modifying the following files with library update,
    node version change, Babel 6 to 7 upgrade ..

    .nvmrc

    「9.9.0」→「12.17.0」

    12.17.0
    

    index.js

    Babel7 compatible
    「require('babel-register');」→「require('@babel/register');」

    require('@babel/register');
    require('src');
    

    .babelrc

    Babel7 compatible

    {
      "presets": [
        "@babel/preset-env"
      ]
    }
    

    package.json

    Babel7 compatible
    npm JavaScript library update

    {
      "name": "@bitsensor/elastalert",
      "version": "3.0.0-beta.0",
      "description": "A server that runs ElastAlert and exposes REST API's for manipulating rules and alerts.",
      "license": "MIT",
      "main": "index.js",
      "author": {
        "name": "BitSensor",
        "url": "https://bitsensor.io",
        "email": "dev@bitsensor.io"
      },
      "repository": {
        "type": "git",
        "url": "git+https://github.com/bitsensor/elastalert.git"
      },
      "directories": {
        "lib": "./lib",
        "test": "./test"
      },
      "dependencies": {
        "@babel/cli": "^7.10.1",
        "@babel/core": "^7.10.1",
        "@babel/preset-env": "^7.10.1",
        "@babel/register": "^7.10.1",
        "body-parser": "^1.19.0",
        "bunyan": "^1.8.12",
        "cors": "^2.8.5",
        "elasticsearch": "^16.7.1",
        "express": "^4.17.1",
        "fs-extra": "^9.0.0",
        "joi": "^14.3.1",
        "lodash": "^4.17.15",
        "mkdirp": "^1.0.4",
        "object-resolve-path": "^1.1.1",
        "randomstring": "^1.1.5",
        "raven": "^2.6.4",
        "request": "^2.88.2",
        "request-promise-native": "^1.0.8",
        "tar": "^6.0.2",
        "ws": "^7.3.0"
      },
      "devDependencies": {
        "eslint": "^7.1.0",
        "husky": "^4.2.5",
        "istanbul": "^0.4.5",
        "mocha": "~7.2.0"
      },
      "scripts": {
        "build": "babel src -d lib",
        "start": "sh ./scripts/start.sh",
        "test": "./scripts/test.sh",
        "update-authors": "./scripts/update-authors.sh",
        "precommit": "./node_modules/eslint/bin/eslint.js ."
      }
    }
    

    Makefile

    「v ?= v0.2.0b2」→「v ?= v0.2.4」
    「docker pull alpine:latest && docker pull node:alpine」→「docker pull alpine:3.11 && docker pull node:14-alpine」
    「docker run -it --rm -p 3030:3030 \」→「docker run -it --rm -p 3030:3030 -p 3333:3333 \」

    v ?= v0.2.4
    
    all: build
    
    build:
    	docker pull alpine:3.11 && docker pull node:14-alpine
    	docker build --build-arg ELASTALERT_VERSION=$(v) -t elastalert .
    
    server: build
    	docker run -it --rm -p 3030:3030 -p 3333:3333 \
    	--net="host" \
    	elastalert:latest
    
    .PHONY: build
    

    docker image create

    nvm install v12.17.0
    nvm use
    docker build -t elastalert:0.2.4 .
    
  2. nsano-rururu commented on May 30, 2020

    @nsano-rururu

    The following information is likely to require further action.

    istanbul

    stanbul is Deprecated.
    https://www.npmjs.com/package/istanbul

    This package has been deprecated
    Author message:

    This module is no longer maintained, try this instead: npm i >nyc Visit https://istanbul.js.org/integrations for other >alternatives.

    <Files that may be relevant>
    ・.gitignore
    ・.npmignore
    ・package.json
    ・scripts/test.sh

    joi

    joi is deprecated, so it needs to be @hapi/joi.

    https://www.npmjs.com/package/joi

    This package has been deprecated
    Author message:

    This module has moved and is now available at @hapi/joi. >Please update your dependencies as this version is no longer >maintained an may contain bugs and security issues.

    @hapi/joi
    https://www.npmjs.com/package/@hapi/joi

    <Files that may be relevant>
    ・package.json
    ・src/common/config/schema.js
    ・src/common/config/server_config.js
    ・src/handlers/test/post.js

    request and request-promise-native

    It became Deprecated on February 11, 2020.

    Alternative libraries to request #3143

    It seems that I'm just using request-promise-native in src/controllers/rules/index.js.

  3. nsano-rururu commented on Jun 27, 2020

    @nsano-rururu

    changed to @hapi/joi as joi is deprecated

    package.json

    delete
    "joi": "^14.3.1",

    add
    "@hapi/joi": "^17.1.1",

    {
      "name": "@bitsensor/elastalert",
      "version": "3.0.0-beta.0",
      "description": "A server that runs ElastAlert and exposes REST API's for manipulating rules and alerts.",
      "license": "MIT",
      "main": "index.js",
      "author": {
        "name": "BitSensor",
        "url": "https://bitsensor.io",
        "email": "dev@bitsensor.io"
      },
      "repository": {
        "type": "git",
        "url": "git+https://github.com/bitsensor/elastalert.git"
      },
      "directories": {
        "lib": "./lib",
        "test": "./test"
      },
      "dependencies": {
        "@babel/cli": "^7.10.1",
        "@babel/core": "^7.10.1",
        "@babel/preset-env": "^7.10.1",
        "@babel/register": "^7.10.1",
        "body-parser": "^1.19.0",
        "bunyan": "^1.8.12",
        "cors": "^2.8.5",
        "elasticsearch": "^16.7.1",
        "express": "^4.17.1",
        "fs-extra": "^9.0.0",
        "@hapi/joi": "^17.1.1",
        "lodash": "^4.17.15",
        "mkdirp": "^0.5.5",
        "object-resolve-path": "^1.1.1",
        "randomstring": "^1.1.5",
        "raven": "^2.6.4",
        "request": "^2.88.2",
        "request-promise-native": "^1.0.8",
        "tar": "^6.0.2",
        "ws": "^7.3.0"
      },
      "devDependencies": {
        "eslint": "^7.1.0",
        "husky": "^4.2.5",
        "istanbul": "^0.4.5",
        "mocha": "~7.2.0"
      },
      "scripts": {
        "build": "babel src -d lib",
        "start": "sh ./scripts/start.sh",
        "test": "./scripts/test.sh",
        "update-authors": "./scripts/update-authors.sh",
        "precommit": "./node_modules/eslint/bin/eslint.js ."
      }
    }
    

    src/common/config/schema.js

    import Joi from 'joi'; → import Joi from '@hapi/joi';

    // Defines the config's Joi schema
    import Joi from '@hapi/joi';
    
    const schema = Joi.object().keys({
      'appName': Joi.string().default('elastalert-server'),
      'es_host': Joi.string().default('elastalert'),
      'es_port': Joi.number().default(9200),
      'writeback_index': Joi.string().default('elastalert_status'),
      'port': Joi.number().default(3030),
      'wsport': Joi.number().default(3333),
      'elastalertPath': Joi.string().default('/opt/elastalert'),
      'rulesPath': Joi.object().keys({
        'relative': Joi.boolean().default(true),
        'path': Joi.string().default('/rules')
      }).default(),
      'templatesPath': Joi.object().keys({
        'relative': Joi.boolean().default(true),
        'path': Joi.string().default('/rule_templates')
      }).default(),
      'dataPath': Joi.object().keys({
        'relative': Joi.boolean().default(true),
        'path': Joi.string().default('/server_data')
      }).default()
    }).default();
    
    export default schema;
    

    src/common/config/server_config.js

    delete
    import Joi from 'joi';
    change
    this._jsonConfig = Joi.validate(jsonConfig, schema).value; → this._jsonConfig = schema.validate(JSON.parse(jsonConfig)).value;

    import fs from 'fs';
    import path from 'path';
    import schema from './schema';
    import resolvePath from 'object-resolve-path';
    import Logger from '../logger';
    
    // Config file relative from project root
    const configFile = 'config/config.json';
    const devConfigFile = 'config/config.dev.json';
    
    const configPath = path.join(process.cwd(), configFile);
    const devConfigPath = path.join(process.cwd(), devConfigFile);
    const logger = new Logger('Config');
    
    export default class ServerConfig {
      constructor() {
        // ready() callbacks
        this._waitList = [];
    
        // Actual config
        this._jsonConfig = null;
      }
    
      /**
       * Get a value from the config.
       *
       * @param {String} key The key to load the value from.
       * @returns {*}
       */
      get(key) {
        return resolvePath(this._jsonConfig, key);
      }
    
      /**
       * Register a callback to call when the config is ready loading.
       *
       * @param {Function} callback The callback to register.
       */
      ready(callback) {
        this._waitList.push(callback);
      }
    
      /**
       * Loads the config by reading the config file or falling back to defaults.
       *
       * @returns {Promise} Returns a promise which resolves when everything is done (as a promise would).
       */
      load() {
    
        //TODO: Watch config file for changes and reload
        const self = this;
        return new Promise(function (resolve) {
          self._getConfig().then(function (config) {
            self._validate(config);
            resolve();
          });
        }).then(function () {
          self._waitList.forEach(function (callback) {
            callback();
          });
        });
      }
    
      _getConfig() {
        const self = this;
    
        return new Promise(function (resolve) {
          self._fileExists(devConfigPath).then(function (devConfigFound) {
    
            // If a dev config was found read it, otherwise check for normal config
            if (devConfigFound) {
              self._readFile(devConfigPath)
                .then(function (config) {
                  resolve(config);
                })
                .catch(function () {
                  resolve({});
                });
            } else {
              logger.info('Proceeding to look for normal config file.');
              self._fileExists(configPath).then(function (configFound) {
                if (configFound) {
                  self._readFile(configPath)
                    .then(function (config) {
                      resolve(config);
                    })
                    .catch(function () {
                      resolve({});
                    });
                } else {
                  logger.info('Using default config.');
                  // If no config was found, return empty object to load defaults
                  resolve({});
                }
              });
            }
          });
        });
      }
    
      /**
       * Checks if the config file exists and we have reading permissions
       *
       * @returns {Promise} Promise returning true if the file was found and false otherwise.
       * @private
       */
      _fileExists(filePath) {
        return new Promise(function (resolve) {
          // Check if the config file exists and has reading permissions
          try {
            fs.access(filePath, fs.F_OK | fs.R_OK, function (error) {
              if (error) {
                if (error.errno === -2) {
                  logger.info(`No ${path.basename(filePath)} file was found in ${filePath}.`);
                } else {
                  logger.warn(`${filePath} can't be read because of reading permission problems. Falling back to default configuration.`);
                }
                resolve(false);
              } else {
                logger.info(`A config file was found in ${filePath}. Using that config.`);
                resolve(true);
              }
            });
          } catch (error) {
            logger.error('Error getting access information with fs using `fs.access`. Error:', error);
          }
        });
      }
    
      /**
       * Reads the config file.
       *
       * @returns {Promise} Promise returning the config if successfully read. Rejects if reading the config failed.
       * @private
       */
      _readFile(file) {
        return new Promise(function (resolve, reject) {
          fs.readFile(file, 'utf8', function (error, config) {
            if (error) {
              logger.warn(`Unable to read config file in (${file}). Using default configuration. Error: `, error);
              reject();
            } else {
              resolve(config);
            }
          });
        });
      }
    
      /**
       * Validate the config using the Joi schema.
       *
       * @param {Object} jsonConfig The config to validate.
       * @private
       */
      _validate(jsonConfig) {
        // Validate the JSON config
        try {
          this._jsonConfig = schema.validate(JSON.parse(jsonConfig)).value;
        } catch (error) {
          logger.error('The config in \'config/config.json\' is not a valid config configuration. Error: ', error);
        }
      }
    }
    

    src/handlers/test/post.js

    import Joi from 'joi'; → import Joi from '@hapi/joi';
    const validationResult = Joi.validate(request.body.options, optionsSchema); → const validationResult = optionsSchema.validate(request.body.options);

    import RouteLogger from '../../routes/route_logger';
    import {sendRequestError} from '../../common/errors/utils';
    import {BodyNotSendError, RuleNotSendError, OptionsInvalidError} from '../../common/errors/test_request_errors';
    import Joi from '@hapi/joi';
    
    let logger = new RouteLogger('/test', 'POST');
    
    const optionsSchema = Joi.object().keys({
      testType: Joi.string().valid('all', 'schemaOnly', 'countOnly').default('all'),
      days: Joi.number().min(1).default(1),
      alert: Joi.boolean().default(false),
      format: Joi.string().default(''),
      maxResults: Joi.number().default(0)
    }).default();
    
    function analyzeRequest(request) {
      if (!request.body) {
        return new BodyNotSendError();
      }
    
      if (!request.body.rule) {
        return new RuleNotSendError();
      }
    
      const validationResult = optionsSchema.validate(request.body.options);
    
      if (validationResult.error) {
        return new OptionsInvalidError(validationResult.error);
      }
    
      let body = request.body;
      body.options = validationResult.value;
    
      return body;
    }
    
    export default function testPostHandler(request, response) {
      /**
       * @type {ElastalertServer}
       */
      let server = request.app.get('server');
      let body = analyzeRequest(request);
    
      if (body.error) {
        logger.sendFailed(body.error);
        sendRequestError(response, body.error);
      }
    
      server.testController.testRule(body.rule, body.options)
        .then(function (consoleOutput) {
          response.send(consoleOutput);
        })
        .catch(function (consoleOutput) {
          response.status(500).send(consoleOutput);
        });
    }
    
  4. nsano-rururu commented on Jun 29, 2020

    @nsano-rururu

    istanbul

    stanbulis Deprecated

    Since there is no test code, coverage tool is not required at this time.
    It would have been introduced and left in place to add.

    Delete File

    scripts/test.sh

    package.json

    delete:「"test": "./scripts/test.sh",」
    delete:「"istanbul": "^0.4.5",」

    {
      "name": "@bitsensor/elastalert",
      "version": "3.0.0-beta.0",
      "description": "A server that runs ElastAlert and exposes REST API's for manipulating rules and alerts.",
      "license": "MIT",
      "main": "index.js",
      "author": {
        "name": "BitSensor",
        "url": "https://bitsensor.io",
        "email": "dev@bitsensor.io"
      },
      "repository": {
        "type": "git",
        "url": "git+https://github.com/bitsensor/elastalert.git"
      },
      "directories": {
        "lib": "./lib",
        "test": "./test"
      },
      "dependencies": {
        "@babel/cli": "^7.10.1",
        "@babel/core": "^7.10.1",
        "@babel/preset-env": "^7.10.1",
        "@babel/register": "^7.10.1",
        "body-parser": "^1.19.0",
        "bunyan": "^1.8.12",
        "cors": "^2.8.5",
        "elasticsearch": "^16.7.1",
        "express": "^4.17.1",
        "fs-extra": "^9.0.0",
        "@hapi/joi": "^17.1.1",
        "lodash": "^4.17.15",
        "mkdirp": "^1.0.4",
        "object-resolve-path": "^1.1.1",
        "randomstring": "^1.1.5",
        "raven": "^2.6.4",
        "request": "^2.88.2",
        "request-promise-native": "^1.0.8",
        "tar": "^6.0.2",
        "ws": "^7.3.0"
      },
      "devDependencies": {
        "eslint": "^7.1.0",
        "husky": "^4.2.5",
        "mocha": "~7.2.0"
      },
      "scripts": {
        "build": "babel src -d lib",
        "start": "sh ./scripts/start.sh",
        "update-authors": "./scripts/update-authors.sh",
        "precommit": "./node_modules/eslint/bin/eslint.js ."
      }
    }
    
  5. nirgilboa commented on Jun 30, 2020

    @nirgilboa
    Author

    @nsano-rururu I'm thinking about forking and starting to maintain this on my own - would you like to make a joint effort?

  6. nirgilboa commented on Jun 30, 2020

    @nirgilboa
    Author

    @nsano-rururu I'm thinking about forking and starting to maintain this on my own - would you like to make a joint effort?

    Yes.

    Cool - I also dropped an email to the maintainer, let's give him a day or two to respond

  7. nirgilboa commented on Jun 30, 2020

    @nirgilboa
    Author

    Ok bitsensor apparently went under and revoked rights for this repo, so we have to continue on a fork.
    Looks like two other contributors already forked and made some changes - let's pick one of the forks and pool our resources on it ?

  8. nirgilboa commented on Jun 30, 2020

    @nirgilboa
    Author

    The repo which you mentioned diverges quite a bit from the master here.. not sure what you meant - do you think that fork is what we should be maintaining ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions