Skip to content

a port of Mime.processDecoded (to MsgBlocks) #1223

Closed

Description

@tomholub

For decoding mime content, in TS we use this method:

  public static decode = async (mimeMsg: Uint8Array): Promise<MimeContent> => {
    let mimeContent: MimeContent = { attachments: [], headers: {}, subject: undefined, text: undefined, html: undefined, signature: undefined, from: undefined, to: [], cc: [], bcc: [] };
    const parser = new MimeParser();
    const leafNodes: { [key: string]: MimeParserNode } = {};
    parser.onbody = (node: MimeParserNode) => {
      const path = String(node.path.join('.'));
      if (typeof leafNodes[path] === 'undefined') {
        leafNodes[path] = node;
      }
    };
    return await new Promise((resolve, reject) => {
      try {
        parser.onend = () => {
          try {
            for (const name of Object.keys(parser.node.headers)) {
              mimeContent.headers[name] = parser.node.headers[name][0].value;
            }
            mimeContent.rawSignedContent = Mime.retrieveRawSignedContent([parser.node]);
            for (const node of Object.values(leafNodes)) {
              if (Mime.getNodeType(node) === 'application/pgp-signature') {
                mimeContent.signature = node.rawContent;
              } else if (Mime.getNodeType(node) === 'text/html' && !Mime.getNodeFilename(node)) {
                // html content may be broken up into smaller pieces by attachments in between
                // AppleMail does this with inline attachments
                mimeContent.html = (mimeContent.html || '') + Mime.getNodeContentAsUtfStr(node);
              } else if (Mime.getNodeType(node) === 'text/plain' && (!Mime.getNodeFilename(node) || Mime.isNodeInline(node))) {
                mimeContent.text = (mimeContent.text ? `${mimeContent.text}\n\n` : '') + Mime.getNodeContentAsUtfStr(node);
              } else if (Mime.getNodeType(node) === 'text/rfc822-headers') {
                if (node._parentNode && node._parentNode.headers.subject) {
                  mimeContent.subject = node._parentNode.headers.subject[0].value;
                }
              } else {
                mimeContent.attachments.push(Mime.getNodeAsAttachment(node));
              }
            }
            const headers = Mime.headerGetAddress(mimeContent, ['from', 'to', 'cc', 'bcc']);
            mimeContent.subject = String(mimeContent.subject || mimeContent.headers.subject || '');
            mimeContent = Object.assign(mimeContent, headers);
            resolve(mimeContent);
          } catch (e) {
            reject(e);
          }
        };
        parser.write(mimeMsg);
        parser.end();
      } catch (e) { // todo - on Android we may want to fail when this happens, evaluate effect on browser extension
        Catch.reportErr(e);
        resolve(mimeContent);
      }
    });
  }

But @DenBond7 probably already has some method for decoding mime. Therefore I don't want to reimplement the above, but instead use Den's methods (and its outputs) for the above purpose (if possible)

Then you take outputs from the MIME decode, and you feed them into a new method that needs implementing, similar to the one below:

  public static processDecoded = (decoded: MimeContent): MimeProccesedMsg => {
    const blocks: MsgBlock[] = [];
    if (decoded.text) {
      const blocksFromTextPart = MsgBlockParser.detectBlocks(Str.normalize(decoded.text)).blocks;
      // if there are some encryption-related blocks found in the text section, which we can use, and not look at the html section
      if (blocksFromTextPart.find(b => b.type === 'encryptedMsg' || b.type === 'signedMsg' || b.type === 'publicKey' || b.type === 'privateKey')) {
        blocks.push(...blocksFromTextPart); // because the html most likely containt the same thing, just harder to parse pgp sections cause it's html
      } else if (decoded.html) { // if no pgp blocks found in text part and there is html part, prefer html
        blocks.push(MsgBlock.fromContent('plainHtml', decoded.html));
      } else { // else if no html and just a plain text message, use that
        blocks.push(...blocksFromTextPart);
      }
    } else if (decoded.html) {
      blocks.push(MsgBlock.fromContent('plainHtml', decoded.html));
    }
    for (const file of decoded.attachments) {
      const treatAs = file.treatAs();
      if (treatAs === 'encryptedMsg') {
        const armored = PgpArmor.clip(file.getData().toUtfStr());
        if (armored) {
          blocks.push(MsgBlock.fromContent('encryptedMsg', armored));
        }
      } else if (treatAs === 'signature') {
        decoded.signature = decoded.signature || file.getData().toUtfStr();
      } else if (treatAs === 'publicKey') {
        blocks.push(...MsgBlockParser.detectBlocks(file.getData().toUtfStr()).blocks);
      } else if (treatAs === 'privateKey') {
        blocks.push(...MsgBlockParser.detectBlocks(file.getData().toUtfStr()).blocks);
      } else if (treatAs === 'encryptedFile') {
        blocks.push(MsgBlock.fromAttachment('encryptedAttachment', '', { name: file.name, type: file.type, length: file.getData().length, data: file.getData() }));
      } else if (treatAs === 'plainFile') {
        blocks.push(MsgBlock.fromAttachment('plainAttachment', '', {
          name: file.name, type: file.type, length: file.getData().length, data: file.getData(), inline: file.inline, cid: file.cid
        }));
      }
    }
    if (decoded.signature) {
      for (const block of blocks) {
        if (block.type === 'plainText') {
          block.type = 'signedText';
          block.signature = decoded.signature;
        } else if (block.type === 'plainHtml') {
          block.type = 'signedHtml';
          block.signature = decoded.signature;
        }
      }
      if (!blocks.find(block => ['plainText', 'plainHtml', 'signedMsg', 'signedHtml', 'signedText'].includes(block.type))) { // signed an empty message
        blocks.push(new MsgBlock("signedMsg", "", true, decoded.signature));
      }
    }
    return { headers: decoded.headers, blocks, from: decoded.from, to: decoded.to, rawSignedContent: decoded.rawSignedContent };
  }

Activity

  1. tomholub commented on May 10, 2021

    @tomholub
    CollaboratorAuthor

    This is an interesting method that needs re-implementing, which will help the app decide how to treat each attachment. Here this is some sort of attachment object that has name, size, mime type and similar meta info on it.

      public treatAs = (): Attachment$treatAs => {
        if (['PGPexch.htm.pgp', 'PGPMIME version identification', 'Version.txt', 'PGPMIME Versions Identification'].includes(this.name)) {
          return 'hidden';  // PGPexch.htm.pgp is html alternative of textual body content produced by PGP Desktop and GPG4o
        } else if (this.name === 'signature.asc' || this.type === 'application/pgp-signature') {
          return 'signature';
        } else if (!this.name && !this.type.startsWith('image/')) { // this.name may be '' or undefined - catch either
          return this.length < 100 ? 'hidden' : 'encryptedMsg';
        } else if (this.name === 'msg.asc' && this.length < 100 && this.type === 'application/pgp-encrypted') {
          return 'hidden'; // mail.ch does this - although it looks like encrypted msg, it will just contain PGP version eg "Version: 1"
        } else if (Attachment.encryptedMsgNames.includes(this.name)) {
          return 'encryptedMsg';
        } else if (this.name.match(/(\.pgp$)|(\.gpg$)|(\.[a-zA-Z0-9]{3,4}\.asc$)/g)) { // ends with one of .gpg, .pgp, .???.asc, .????.asc
          return 'encryptedFile';
        } else if (this.name.match(/(cryptup|flowcrypt)-backup-[a-z0-9]+\.(key|asc)$/g)) {
          return 'privateKey';
        } else if (this.type === 'application/pgp-keys') {
          return 'publicKey';
        } else if (this.name.match(/^(0|0x)?[A-F0-9]{8}([A-F0-9]{8})?.*\.asc$/g)) { // name starts with a key id
          return 'publicKey';
        } else if (this.name.toLowerCase().includes('public') && this.name.match(/[A-F0-9]{8}.*\.asc$/g)) { // name contains the word "public", any key id and ends with .asc
          return 'publicKey';
        } else if (this.name.match(/\.asc$/) && this.hasData() && Buf.with(this.getData().subarray(0, 100)).toUtfStr().includes('-----BEGIN PGP PUBLIC KEY BLOCK-----')) {
          return 'publicKey';
        } else if (this.name.match(/\.asc$/) && this.length < 100000 && !this.inline) {
          return 'encryptedMsg';
        } else {
          return 'plainFile';
        }
      }
  2. added this to the 1.1.8 milestone on May 10, 2021
  3. DenBond7 commented on May 10, 2021

    @DenBond7
    Collaborator

    @IvanPizhenko Have you worked with JavaMail before?

  4. DenBond7 commented on May 10, 2021

    @DenBond7
    Collaborator

    @tomholub I've looked at TS code. It seems that your logic is clear and can be transferred to Kotlin. There is a lot of work with JavaMail. How I see that

    1. Parse inputStream to MimeMessage
    2. Analyze the parsed message. Part by part. @IvanPizhenko Please look for example at the following
      private suspend fun findEncryptedPart(part: Part): Uri? = withContext(Dispatchers.Default) {

      here you will find how I iterate over the parts. Please feel free to look at other examples over the app.
    3. During analyzing we can decrypt found parts and generate MsgBlock
    4. As I understand in the end we should receive something similar to
    data class ParseDecryptedMsgResult constructor(
        @Expose val text: String?,
        @Expose val replyType: String,
        @Expose val subject: String?,
        @SerializedName("error")
        @Expose override val apiError: ApiError?, // here we should have decryptionError
        var msgBlocks: MutableList<MsgBlock>?) {}
    
  5. IvanPizhenko commented on May 11, 2021

    @IvanPizhenko
    Contributor

    @IvanPizhenko Have you worked with JavaMail before?

    a bit, I've implemented some sending emails in the FES.

  6. IvanPizhenko commented on May 11, 2021

    @IvanPizhenko
    Contributor

    @tomholub So which one I should continue with now - this one or #1057 ?

  7. tomholub commented on May 11, 2021

    @tomholub
    CollaboratorAuthor

    This one please. It's a prerequisite for the other one.

  8. IvanPizhenko commented on May 11, 2021

    @IvanPizhenko
    Contributor

    ok

  9. IvanPizhenko commented on May 23, 2021

    @IvanPizhenko
    Contributor

    @tomholub where can I get some Typescript unit tests for this?

  10. tomholub commented on May 23, 2021

    @tomholub
    CollaboratorAuthor

    I'll extract some test cases for you

  11. IvanPizhenko commented on May 23, 2021

    @IvanPizhenko
    Contributor

    And question in advance to @DenBond7 - creating MimeMessage object requires Session object. How can I obtain it for unit tests?

  12. 11 remaining items

  13. DenBond7 commented on May 24, 2021

    @DenBond7
    Collaborator

    @IvanPizhenko I think MimeContent is redundant. JavaMail covers it with MimeMessage. You don't need to parse MIME message manually, just use JavaMail and MimeMessage as a parsed object.

    @Suppress("ArrayInDataClass")
      data class MimeContent(
        val attachments: List<BodyPart>,
        var signature: ByteArray?,
        val subject: String,
        val html: String?,
        val text: String?,
        val from: Array<Address>?,
        val to: Array<Address>?,
        val cc: Array<Address>?,
        val bcc: Array<Address>?
      )
    
  14. tomholub commented on May 24, 2021

    @tomholub
    CollaboratorAuthor

    @DenBond7 how about signedText and signedHtml? This may also be returned by current node - what would happen to them?

  15. tomholub commented on May 24, 2021

    @tomholub
    CollaboratorAuthor

    Or maybe you don't even send them to node, since they are not encrypted?

  16. DenBond7 commented on May 24, 2021

    @DenBond7
    Collaborator

    @tomholub Could you clarify a question? Too many 'node' words, I'm not sure that I understand right according to the context.

  17. tomholub commented on May 24, 2021

    @tomholub
    CollaboratorAuthor

    We also have MsgBlock types signedHtml and signedText that may be returned by NodeJS (I saw it in some of the test cases above). There's also publicKey for example. How are these handled on Android?

  18. DenBond7 commented on May 24, 2021

    @DenBond7
    Collaborator

    Thank you. Got it.

    Please look at the follwoing code:

    This method contains a logic where we handle MsgBlock

  19. tomholub commented on May 24, 2021

    @tomholub
    CollaboratorAuthor

    I see - you treat signedText and signedHtml as "other" and render them as text (which would be a bug in the case of signedHtml).

    If that is the case, Ivan's still has to be able to process most of the test cases that I submitted above, else these blocks would not show up, even to be rendered as "other".

    @IvanPizhenko see what level of compatibility can you achieve easily. Then we can discuss if we need to support the remaining tests or not yet.

  20. IvanPizhenko commented on May 30, 2021

    @IvanPizhenko
    Contributor

    @tomholub @DenBond7 Do we need the rawSignedContent? In the Typescript (in the decode() listed above) it is extracted as follows:

    mimeContent.rawSignedContent = Mime.retrieveRawSignedContent([parser.node]);

    I did not port this, because I am not sure if this is needed. Otherwise I almost have it ready and can post a PR soon.

  21. added 2 commits that reference this issue on May 30, 2021
  22. DenBond7 commented on May 31, 2021

    @DenBond7
    Collaborator

    Do we need the rawSignedContent?

    As we don't support handling the signed content yet I think it's redundant for now. it can be done as improvements

  23. IvanPizhenko commented on May 31, 2021

    @IvanPizhenko
    Contributor

    ok

  24. added a commit that references this issue on May 31, 2021
  25. added a commit that references this issue on May 31, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions